{"id":10996,"date":"2019-12-04T14:00:48","date_gmt":"2019-12-04T19:00:48","guid":{"rendered":"https:\/\/qxf2.com\/blog\/?p=10996"},"modified":"2019-12-04T09:17:26","modified_gmt":"2019-12-04T14:17:26","slug":"testing-with-celery","status":"publish","type":"post","link":"https:\/\/qxf2.com\/blog\/testing-with-celery\/","title":{"rendered":"Testing with Celery"},"content":{"rendered":"<p>As part of the pairing project activity at <a href=\"https:\/\/qxf2.com\/\">Qxf2<\/a> &#8211; I had to demonstrate a simple working example that included Celery and it&#8217;s monitoring tool called Flower. I knew a little bit of background as I have tested a product that includes Celery.<br \/>\nI read the Celery guide, a few related articles, and simple examples. I also came across the asynchronous behavior of Celery. I wanted to come up with a working example that connects the tech involved i.e. Celery, Redis, RabbitMQ, and Flower, and hence this post.<br \/>\nThis post will introduce you to Celery, the other basic tech required to set up a working example. The example shows the asynchronous behavior of Celery.<\/p>\n<hr \/>\n<p>This post has these sections:<br \/>\n1. What is Celery?<br \/>\n2. What is Flower?<br \/>\n3. A brief about the example<br \/>\n4. A step by step guide<\/p>\n<h3>What is Celery?<\/h3>\n<ul>\n<li>Celery is an asynchronous task queue based on distributed message passing.<\/li>\n<li>Celery system can consist of a single worker, multiple workers, and brokers, giving way to high availability and horizontal scaling.<\/li>\n<li>Celery communicates via messages, usually using a broker to mediate between clients and workers.<\/li>\n<li>Celery is written in Python, but the protocol can be implemented in any language.<\/li>\n<li>Tasks can execute asynchronously (in the background) or synchronously (wait until ready).<\/li>\n<li>The recommended message brokers are RabbitMQ or Redis. Additionally, MongoDB, Beanstalk, Amazon SQS, CouchDB, IronMQ, and databases (using SQLAlchemy or the Django ORM) are supported in status experimental<\/li>\n<\/ul>\n<p>For more information, click <a href=\"https:\/\/docs.celeryproject.org\/en\/latest\/userguide\/\">here<\/a>.<\/p>\n<hr \/>\n<h3>What is Flower?<\/h3>\n<ul>\n<li>Flower is a web-based tool for monitoring and administrating Celery clusters.<\/li>\n<li>You can also monitor the progress of the Celery tasks, view graphs and statistics.<\/li>\n<\/ul>\n<p>For more information, click <a href=\"https:\/\/flower.readthedocs.io\/en\/latest\/index.html\">here<\/a>.<\/p>\n<hr \/>\n<h3>A brief about the example:<\/h3>\n<p>In this example, I am demonstrating asynchronous processing functionality using Celery i.e. how we can execute tasks in the background without waiting for the task to finish. We can also view these tasks in the Flower dashboard.<br \/>\n<em>Please note that it&#8217;s preferable to use a Linux machine for this example.<\/em><\/p>\n<hr \/>\n<h3>A step by step guide:<\/h3>\n<p><strong>STEP 1. Installation<\/strong><br \/>\nHere we&#8217;ll ssh into the Linux machine and install Celery, RabbitMQ and Flower.<br \/>\na. Install Celery using<\/p>\n<pre lang=\"python\">pip install celery<\/pre>\n<p>b. Get set up with RabbitMQ by following the section &#8220;Setting up RabbitMQ&#8221; from this <a href=\"https:\/\/docs.celeryproject.org\/en\/latest\/getting-started\/brokers\/rabbitmq.html#broker-rabbitmq\">link <\/a><br \/>\nc. Install Flower using<\/p>\n<pre lang=\"python\">pip install flower<\/pre>\n<p><strong>STEP 2. Project setup<\/strong><br \/>\nThis section tells you about the structure of the directory and also explains the content of the files within the directory which are needed for the example. Create the following structure:<\/p>\n<pre>test_celery\r\n    __init__.py\r\n    celeryconfig.py\r\n    run_tasks.py\r\n    tasks.py\r\n<\/pre>\n<p>a. <em>test_celery<\/em> is the parent directory which consists of all the listed files above.<br \/>\nb. <em>celeryconfig.py<\/em> will have the broker(RabbitMQ) and backend(Redis) result details.<\/p>\n<pre lang=\"python\">BROKER_URL = 'amqp:\/\/username:password@localhost:5672\/myvhost'\r\nbroker_url = 'redis:\/\/username@localhost:6379\/0'\r\nCELERY_IMPORTS = ('tasks', )\r\n<\/pre>\n<p>c. <em>tasks.py<\/em> Here let&#8217;s initialize the &#8216;app&#8217; using Celery with the broker and backend details. Create 2 methods, one for add and the other for minus. Add appropriate print statements so that we know which method is being called.<\/p>\n<pre lang=\"python\">from celery import Celery\r\n\r\nimport time\r\napp = Celery('tasks', broker='pyamqp:\/\/username:password@localhost:5672\/vhost', backend='redis:\/\/localhost')\r\n\r\n@app.task\r\ndef add(x, y):\r\n    print(\"task 1 working here\")\r\n    return x + y\r\n\r\n@app.task\r\ndef minus(x, y):\r\n    print(\"task 2 working here\")\r\n    return x - y\r\n<\/pre>\n<p>d. <em>run_tasks.py<\/em> has 2 blocks, one for each task\/worker where the &#8216;add&#8217; and &#8216;minus&#8217; methods\/tasks are called. delay method is used for asynchronous processing. Once the &#8216;run_tasks.py&#8217; is executed, you would see some useful information about the status of each task and the result as well. Initially, when the task has not been finished, it returns a None. Once the task is complete, it returns the result. Task1 ends and Task2 starts immediately. Task2 works in a similar way Task1 functioned.<\/p>\n<pre lang=\"python\">from tasks import add\r\nfrom tasks import minus\r\nimport time\r\n\r\nif __name__ == '__main__':\r\n    result = add.delay(5,5)\r\n    # at this time, our task is not finished, so it will return False\r\n    print('Has Task1 finished?'), result.ready()\r\n    print('What is the Task1 result: '), result.result\r\n    # sleep 10 seconds to ensure the task has been finished\r\n    time.sleep(10)\r\n    # now the task should be finished and ready method will return True\r\n    print('Has Task 1 finished?'), result.ready()\r\n    print('What is the Task 1 completed result:'), result.result\r\n\r\n    result = minus.delay(20,10)\r\n    # at this time, our task is not finished, so it will return False\r\n    print('Has Task2 finished?'), result.ready()\r\n    print('What is the Task2 result:'), result.result\r\n    # sleep 10 seconds to ensure the task has been finished\r\n    time.sleep(10)\r\n    # now the task should be finished and ready method will return True\r\n    print('Has Task 2 finished? '), result.ready()\r\n    print('What is the Task 2 completed result: '), result.result\r\n<\/pre>\n<p><strong>STEP 3. Start Celery, RabbitMQ, Flower<\/strong><br \/>\nNow that the project has been set up, let&#8217;s start Celery, RabbitMQ, and Flower:<br \/>\na. Start RabbitMQ server using the command<\/p>\n<pre lang=\"python\">rabbitmq-server<\/pre>\n<p>b. Start Celery workers: On two separate consoles, start Celery worker using the command below (run in the parent folder of our project folder <em>test_celery<\/em>):<br \/>\nOn one console:<\/p>\n<pre lang=\"python\">celery -A tasks worker --loglevel=INFO --concurrency=10 -n worker1@guest1<\/pre>\n<p>On another console:<\/p>\n<pre lang=\"python\">celery -A tasks worker --loglevel=INFO --concurrency=10 -n worker2@guest2<\/pre>\n<p>Celery successfully connects to RabbitMQ:<br \/>\nThis is the image of Worker1<br \/>\n<a href=\"https:\/\/qxf2.com\/blog\/wp-content\/uploads\/2020\/11\/worker1.jpg\" data-rel=\"lightbox-image-0\" data-rl_title=\"\" data-rl_caption=\"\" title=\"\"><img loading=\"lazy\" decoding=\"async\" class=\"alignleft size-full wp-image-11678\" src=\"https:\/\/qxf2.com\/blog\/wp-content\/uploads\/2020\/11\/worker1.jpg\" alt=\"Worker1\" width=\"904\" height=\"374\" srcset=\"https:\/\/qxf2.com\/blog\/wp-content\/uploads\/2020\/11\/worker1.jpg 904w, https:\/\/qxf2.com\/blog\/wp-content\/uploads\/2020\/11\/worker1-300x124.jpg 300w, https:\/\/qxf2.com\/blog\/wp-content\/uploads\/2020\/11\/worker1-768x318.jpg 768w\" sizes=\"auto, (max-width: 904px) 100vw, 904px\" \/><\/a><\/p>\n<p>This is the image of Worker2<br \/>\n<a href=\"https:\/\/qxf2.com\/blog\/wp-content\/uploads\/2020\/11\/worker2.jpg\" data-rel=\"lightbox-image-1\" data-rl_title=\"\" data-rl_caption=\"\" title=\"\"><img loading=\"lazy\" decoding=\"async\" class=\"alignleft size-full wp-image-11680\" src=\"https:\/\/qxf2.com\/blog\/wp-content\/uploads\/2020\/11\/worker2.jpg\" alt=\"Worker2\" width=\"904\" height=\"397\" srcset=\"https:\/\/qxf2.com\/blog\/wp-content\/uploads\/2020\/11\/worker2.jpg 904w, https:\/\/qxf2.com\/blog\/wp-content\/uploads\/2020\/11\/worker2-300x132.jpg 300w, https:\/\/qxf2.com\/blog\/wp-content\/uploads\/2020\/11\/worker2-768x337.jpg 768w\" sizes=\"auto, (max-width: 904px) 100vw, 904px\" \/><\/a><\/p>\n<p>c. Now on another console, start Flower:<\/p>\n<pre lang=\"python\">celery -A tasks flower<\/pre>\n<p>Successful connection to Flower is seen as given below:<\/p>\n<p><a href=\"https:\/\/qxf2.com\/blog\/wp-content\/uploads\/2020\/11\/flower.jpg\" data-rel=\"lightbox-image-2\" data-rl_title=\"\" data-rl_caption=\"\" title=\"\"><img decoding=\"async\" class=\"alignleft size-full\" src=\"https:\/\/qxf2.com\/blog\/wp-content\/uploads\/2020\/11\/flower.jpg\" alt=\"Flower\" width=\"100%\" \/><\/a><\/p>\n<p><strong>STEP 4. Run the python file<\/strong><br \/>\nNow, run the python task &#8220;run_tasks.py&#8221; like this:<\/p>\n<pre lang=\"python\">python run_tasks<\/pre>\n<p>Look at the guest1 celery console now, you&#8217;ll see worker1 received the task and the result 10 is displayed immediately. It prints the task name with task id in brackets. This shows the asynchronous functionality of Celery.<\/p>\n<p>Refer to the below images for Worker1 and Worker2 results.<br \/>\n<a href=\"https:\/\/qxf2.com\/blog\/wp-content\/uploads\/2020\/11\/worker1_results.jpg\" data-rel=\"lightbox-image-3\" data-rl_title=\"\" data-rl_caption=\"\" title=\"\"><img loading=\"lazy\" decoding=\"async\" class=\"alignleft size-full wp-image-11675\" src=\"https:\/\/qxf2.com\/blog\/wp-content\/uploads\/2020\/11\/worker1_results.jpg\" alt=\"Worker1 results\" width=\"1088\" height=\"443\" srcset=\"https:\/\/qxf2.com\/blog\/wp-content\/uploads\/2020\/11\/worker1_results.jpg 1088w, https:\/\/qxf2.com\/blog\/wp-content\/uploads\/2020\/11\/worker1_results-300x122.jpg 300w, https:\/\/qxf2.com\/blog\/wp-content\/uploads\/2020\/11\/worker1_results-768x313.jpg 768w, https:\/\/qxf2.com\/blog\/wp-content\/uploads\/2020\/11\/worker1_results-1024x417.jpg 1024w\" sizes=\"auto, (max-width: 1088px) 100vw, 1088px\" \/><\/a><\/p>\n<p><a href=\"https:\/\/qxf2.com\/blog\/wp-content\/uploads\/2020\/11\/worker2_results.jpg\" data-rel=\"lightbox-image-4\" data-rl_title=\"\" data-rl_caption=\"\" title=\"\"><img loading=\"lazy\" decoding=\"async\" class=\"alignleft size-full wp-image-11676\" src=\"https:\/\/qxf2.com\/blog\/wp-content\/uploads\/2020\/11\/worker2_results.jpg\" alt=\"Worker2_results\" width=\"1113\" height=\"452\" srcset=\"https:\/\/qxf2.com\/blog\/wp-content\/uploads\/2020\/11\/worker2_results.jpg 1113w, https:\/\/qxf2.com\/blog\/wp-content\/uploads\/2020\/11\/worker2_results-300x122.jpg 300w, https:\/\/qxf2.com\/blog\/wp-content\/uploads\/2020\/11\/worker2_results-768x312.jpg 768w, https:\/\/qxf2.com\/blog\/wp-content\/uploads\/2020\/11\/worker2_results-1024x416.jpg 1024w\" sizes=\"auto, (max-width: 1113px) 100vw, 1113px\" \/><\/a><\/p>\n<p>In the current console where we ran the run_tasks.py, you&#8217;ll see the print statement initially, and then after 10 seconds, you&#8217;ll see the Task1 completed.\u00a0Similarly, Task2 starts and completes by displaying the print statements.<\/p>\n<p><a href=\"https:\/\/qxf2.com\/blog\/wp-content\/uploads\/2020\/11\/run_tasks_results.jpg\" data-rel=\"lightbox-image-5\" data-rl_title=\"\" data-rl_caption=\"\" title=\"\"><img loading=\"lazy\" decoding=\"async\" class=\"alignleft size-full wp-image-11674\" src=\"https:\/\/qxf2.com\/blog\/wp-content\/uploads\/2020\/11\/run_tasks_results.jpg\" alt=\"run_tasks results\" width=\"473\" height=\"172\" srcset=\"https:\/\/qxf2.com\/blog\/wp-content\/uploads\/2020\/11\/run_tasks_results.jpg 473w, https:\/\/qxf2.com\/blog\/wp-content\/uploads\/2020\/11\/run_tasks_results-300x109.jpg 300w\" sizes=\"auto, (max-width: 473px) 100vw, 473px\" \/><\/a><\/p>\n<p>&nbsp;<\/p>\n<p>&nbsp;<\/p>\n<p>&nbsp;<\/p>\n<p>&nbsp;<\/p>\n<p>&nbsp;<\/p>\n<p>&nbsp;<\/p>\n<p><strong>STEP 5. Access Flower<\/strong><br \/>\nNow, access the web console at <strong>http:\/\/localhost:5555<\/strong>, you would see these tasks in worker1 and worker2.<br \/>\n<a href=\"https:\/\/qxf2.com\/blog\/wp-content\/uploads\/2020\/11\/flower_results.jpg\" data-rel=\"lightbox-image-6\" data-rl_title=\"\" data-rl_caption=\"\" title=\"\"><img loading=\"lazy\" decoding=\"async\" class=\"alignleft size-full wp-image-11673\" src=\"https:\/\/qxf2.com\/blog\/wp-content\/uploads\/2020\/11\/flower_results.jpg\" alt=\"Flower results\" width=\"1366\" height=\"407\" srcset=\"https:\/\/qxf2.com\/blog\/wp-content\/uploads\/2020\/11\/flower_results.jpg 1366w, https:\/\/qxf2.com\/blog\/wp-content\/uploads\/2020\/11\/flower_results-300x89.jpg 300w, https:\/\/qxf2.com\/blog\/wp-content\/uploads\/2020\/11\/flower_results-768x229.jpg 768w, https:\/\/qxf2.com\/blog\/wp-content\/uploads\/2020\/11\/flower_results-1024x305.jpg 1024w\" sizes=\"auto, (max-width: 1366px) 100vw, 1366px\" \/><\/a><\/p>\n<hr \/>\n<p>I hope this blog has helped you get set up quickly with Celery and Flower.<\/p>\n<p><strong>References:<\/strong><br \/>\n1. <a href=\"https:\/\/docs.celeryproject.org\/en\/latest\/index.html \">Celery<\/a><\/p>\n<hr \/>\n","protected":false},"excerpt":{"rendered":"<p>As part of the pairing project activity at Qxf2 &#8211; I had to demonstrate a simple working example that included Celery and it&#8217;s monitoring tool called Flower. I knew a little bit of background as I have tested a product that includes Celery. I read the Celery guide, a few related articles, and simple examples. I also came across the [&hellip;]<\/p>\n","protected":false},"author":13,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[203,204,18,206,205],"tags":[192,200,201,202],"class_list":["post-10996","post","type-post","status-publish","format-standard","hentry","category-celery","category-flower","category-python","category-rabbitmq","category-redis","tag-celery","tag-flower","tag-rabbitmq","tag-redis"],"_links":{"self":[{"href":"https:\/\/qxf2.com\/blog\/wp-json\/wp\/v2\/posts\/10996","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/qxf2.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/qxf2.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/qxf2.com\/blog\/wp-json\/wp\/v2\/users\/13"}],"replies":[{"embeddable":true,"href":"https:\/\/qxf2.com\/blog\/wp-json\/wp\/v2\/comments?post=10996"}],"version-history":[{"count":49,"href":"https:\/\/qxf2.com\/blog\/wp-json\/wp\/v2\/posts\/10996\/revisions"}],"predecessor-version":[{"id":11902,"href":"https:\/\/qxf2.com\/blog\/wp-json\/wp\/v2\/posts\/10996\/revisions\/11902"}],"wp:attachment":[{"href":"https:\/\/qxf2.com\/blog\/wp-json\/wp\/v2\/media?parent=10996"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/qxf2.com\/blog\/wp-json\/wp\/v2\/categories?post=10996"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/qxf2.com\/blog\/wp-json\/wp\/v2\/tags?post=10996"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}