{"id":8075,"date":"2018-01-12T06:12:33","date_gmt":"2018-01-12T11:12:33","guid":{"rendered":"https:\/\/qxf2.com\/blog\/?p=8075"},"modified":"2022-09-11T11:47:40","modified_gmt":"2022-09-11T15:47:40","slug":"getting-started-wallaroo","status":"publish","type":"post","link":"https:\/\/qxf2.com\/blog\/getting-started-wallaroo\/","title":{"rendered":"Getting started with Wallaroo"},"content":{"rendered":"<p>We used to think writing (near) real-time applications that process multiple data streams was for the high IQ crowd and well-funded teams. This belief was probably strengthened by the fact that we (at <a href=\"https:\/\/www.qxf2.com\/?utm_source=wallaroo-start&amp;utm_medium=click&amp;utm_campaign=From%20blog\">Qxf2<\/a>) love Python &#8230; and our favorite language lacked good complex event processors, stream processors. So we were excited to discover <a href=\"http:\/\/www.wallaroolabs.com\/\">Wallaroo<\/a> has set out to fill that gap.<\/p>\n<p>Wallaroo describes itself as &#8220;a fast, elastic data processing engine that rapidly takes you from prototype to production by eliminating infrastructure complexity&#8221;. The charm for us lay in the fact that it could do stream processing and gave us an easy way to designate nodes, write business logic for them and define the structure of a distributed, stream processing application.<\/p>\n<p>We decided to explore Wallaroo. We used a Docker image with sample applications that Wallaroo Labs published. We were able to do the tutorials fairly easily. In this post, we will walk you through one of the examples Wallaroo Labs has provided.<\/p>\n<h3>Wait! Are we using somebody else&#8217;s code for a post?<\/h3>\n<p>Yeah, we will be using Wallaroo Labs&#8217;s Docker image and their own code in this post. So no new code from our end. We know that seems lazy but we have a few good reasons:<\/p>\n<p>a) The excellent posts on <a href=\"https:\/\/blog.wallaroolabs.com\">Wallaroo Labs&#8217; blog<\/a> are written for a far more technical audience. But we know there is a class of engineers who want to first see something work before they actually spend time learning something new. This post is meant for such engineers.<\/p>\n<p>b) Google is struggling to rank Wallaroo Labs&#8217; material well. So there is no simple &#8220;30-minute guide to Wallaroo&#8221; that a reader can scan and grok without having to try things out.<\/p>\n<p>c) This post helped us articulate our understanding (or lack thereof!) of Wallaroo<\/p>\n<hr \/>\n<h3>Getting started with Wallaroo using Docker<\/h3>\n<p>In this blog, we will show you how to set up Wallaroo environment using Docker image and then run a sample &#8220;Celsius to Fahrenheit&#8221; application. We will be using a Linux box. Below are the steps involved<\/p>\n<p>A) Install Docker Community Edition(CE)<br \/>\nB) Get the official Wallaroo Docker image<br \/>\nC) Run a Wallaroo application in Docker<br \/>\nD) Start the Metrics UI<br \/>\nE) Run Giles Receiver<br \/>\nF) Run the &#8220;Celsius to Fahrenheit&#8221; application<br \/>\nG) Send Data to the application<br \/>\nH) Open &#8220;Celsius To Fahrenheit&#8221; application<br \/>\nI) Shut down the Cluster<br \/>\nJ) Shut down Giles Sender and Giles Receiver<br \/>\nK) Shut down the Metrics UI<br \/>\nL) Shut down the Wallaroo container<\/p>\n<hr \/>\n<p><strong>A) Install Docker Community Edition(CE)<\/strong><\/p>\n<p>Get <a href=\"https:\/\/docs.docker.com\/engine\/installation\/\">Docker Community Edition (CE)<\/a> to set up Docker in your system so that we can install the Wallaroo Docker image.<\/p>\n<p><strong>B) Get the official Wallaroo image<\/strong><\/p>\n<p>Once Docker is installed on your system pull the official Wallaroo image.<\/p>\n<p>Open a new terminal and run the following command.<\/p>\n<pre>docker pull wallaroo-labs-docker-wallaroolabs.bintray.io\/release\/wallaroo:0.3.1<\/pre>\n<p>Wallaroo Labs uses a number of names\/words to describe the various components of Wallaroo. We are beginners but we will take a shot at giving you some sort of idea (even if it is inaccurate) about the different components. Following things are included in the Docker image :<\/p>\n<ul>\n<li><strong>Machida:<\/strong> this is Wallaroo&#8217;s run time Python environment. Technology that makes it easy to write applications that handle distributed data streams understandably needs its own run-time environment.<\/li>\n<li><strong>Giles Sender:<\/strong> mimics an incoming data stream and supplies data to Wallaroo applications over TCP.<\/li>\n<li><strong>Giles Receiver:<\/strong> mimics a data sink and receives data from Wallaroo over TCP.<\/li>\n<li><strong>Cluster Shutdown tool:<\/strong> notifies the cluster to shut down cleanly.<\/li>\n<li><strong>Metrics UI:<\/strong> receives and displays metrics for running Wallaroo applications.<\/li>\n<li><strong>Wallaroo Source Code:<\/strong> full source code is provided, including Python example applications.<\/li>\n<\/ul>\n<p><strong>C) Run a Wallaroo application in Docker<\/strong><\/p>\n<p>Run the Wallaroo application in docker by using the following command in new terminal.<\/p>\n<pre>docker run --rm -it --privileged -p 4000:4000 \\\r\n-v \/tmp\/wallaroo-docker\/wallaroo-src:\/src\/wallaroo \\\r\n-v \/tmp\/wallaroo-docker\/python-virtualenv:\/src\/python-virtualenv \\\r\n--name wally \\\r\nwallaroo-labs-docker-wallaroolabs.bintray.io\/release\/wallaroo:0.3.1<\/pre>\n<p><strong>D) Start the Metrics UI<\/strong><\/p>\n<p>Start the Metrics UI to receive and display metrics for running Wallaroo application. Open a new terminal and run the following commands.<\/p>\n<p>1. Enter the Wallaroo Docker container<\/p>\n<pre>docker exec -it wally environment-setup.sh<\/pre>\n<p><span>2. Start the Metrics UI<\/span><\/p>\n<pre><span>metrics_reporter_ui start\r\n<\/span><\/pre>\n<p><span> <\/span> Verify it started up correctly by visiting <a href=\"http:\/\/localhost:4000\/\">http:\/\/localhost:4000<\/a><\/p>\n<p><strong>E) Run Giles Receiver<\/strong><\/p>\n<p>Run Giles Receiver to receives data from Wallaroo over TCP. Open a new terminal and run the following commands.<\/p>\n<p>1. Enter the Wallaroo Docker container<\/p>\n<pre>docker exec -it wally environment-setup.sh<\/pre>\n<p>2. listen for data from Wallaroo application<\/p>\n<pre>receiver --listen 127.0.0.1:5555 --no-write --ponythreads=1 --ponynoblock<\/pre>\n<p>You should see the line <strong>Listening for data<\/strong> that indicates that Giles receiver is up and running.<\/p>\n<p><strong>F) Run the &#8220;Celsius to Fahrenheit&#8221; application<\/strong><\/p>\n<p>To run the &#8220;Celsius to Fahrenheit&#8221; application. This is a stateless application that takes a floating point Celsius value and sends out a floating point Fahrenheit value.Open a new terminal and run the following commands.<\/p>\n<p>1. Enter the Wallaroo Docker container<\/p>\n<pre>docker exec -it wally environment-setup.sh<\/pre>\n<p>2. Go to the python Celsius example directory<\/p>\n<pre>cd \/src\/wallaroo\/examples\/python\/celsius<\/pre>\n<p>3. Run the celsius to fahrenheit application<\/p>\n<pre>machida --application-module celsius --in 127.0.0.1:7000 \\\r\n --out 127.0.0.1:5555 --metrics 127.0.0.1:5001 --control 127.0.0.1:6000 \\\r\n --data 127.0.0.1:6001 --name worker-name --external 127.0.0.1:5050 \\\r\n --cluster-initializer --ponythreads=1 --ponynoblock<\/pre>\n<p>This tells the &#8220;Celsius to Fahrenheit&#8221; application that it should listen on port 7000 for incoming data, write outgoing data to port 5555, and send metrics data to port 5001.<\/p>\n<p><strong>G) Send Data to the application<\/strong><\/p>\n<p>To send the data to the application, open a new terminal and run the following commands.<\/p>\n<p>1. Enter the Wallaroo Docker container<\/p>\n<pre>docker exec -it wally environment-setup.sh<\/pre>\n<p>2. Start the sender with the following command<\/p>\n<pre>sender --host 127.0.0.1:7000 --messages 25000000 --binary \\\r\n--batch-size 50 --interval 10_000_000 --repeat --no-write \\\r\n--msg-size 8 --ponythreads=1 --ponynoblock \\\r\n--file \/src\/wallaroo\/examples\/python\/celsius\/celsius.msg<\/pre>\n<p>A pre-generated data file will repeatedly send messages via Giles Sender until application reach 25,000,000 messages.<\/p>\n<p>If the sender is working correctly, you should see <strong>Connected<\/strong> printed to the screen.<\/p>\n<p><strong>H) Open &#8220;Celsius To Fahrenheit&#8221; application<\/strong><\/p>\n<p>To open the application &#8220;Celsius To Fahrenheit&#8221; in browser visit <a href=\"http:\/\/localhost:4000\">http:\/\/localhost:4000<\/a>. You can look at different metrics related to pipeline, worker and computations by clicking on the hyperlinks. The metric stats will get updated as data is processed through the application<\/p>\n<p><strong>I) Shut down the Cluster<\/strong><\/p>\n<p>To shut down the cluster cleanly, open a new terminal and run the following commands.<\/p>\n<p>1. Enter the Wallaroo Docker container<\/p>\n<pre>docker exec -it wally environment-setup.sh<\/pre>\n<p>2. Shut down the cluster<\/p>\n<pre>cluster_shutdown 127.0.0.1:5050<\/pre>\n<p><strong>J) Shut down Giles Sender and Giles Receiver <\/strong><\/p>\n<p>Press <strong>Ctrl-c<\/strong> from Giles Sender and Giles Receiver shells.<\/p>\n<p><strong>K) Shut down the Metrics UI <\/strong><\/p>\n<pre>metrics_reporter_ui stop<\/pre>\n<p><strong>L) Shut down the Wallaroo container<\/strong><\/p>\n<pre>docker stop wally\r\n<\/pre>\n<hr \/>\n<p>Getting set up with Wallaroo using Docker was fairly simple. Our next step is to try and build an application using Wallaroo. Stay tuned for further updates&#8230;<\/p>\n<p><strong>If you liked what you read, know more <a href=\"https:\/\/qxf2.com\/blog\/about-qxf2\/\">about Qxf2<\/a>.<\/strong><\/p>\n<hr>\n<h3>References:<\/h3>\n<p>Here are some useful references which we followed.<br \/>\n1. <a href=\"https:\/\/docs.wallaroolabs.com\/book\/getting-started\/docker-setup.html\">Setting Up Your Environment for Wallaroo in Docker<\/a><br \/>\n2. <a href=\"https:\/\/docs.wallaroolabs.com\/book\/getting-started\/run-a-wallaroo-application.html\">Run a Wallaroo Application<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>We used to think writing (near) real-time applications that process multiple data streams was for the high IQ crowd and well-funded teams. This belief was probably strengthened by the fact that we (at Qxf2) love Python &#8230; and our favorite language lacked good complex event processors, stream processors. So we were excited to discover Wallaroo has set out to fill [&hellip;]<\/p>\n","protected":false},"author":3,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[140,18,154],"tags":[],"class_list":["post-8075","post","type-post","status-publish","format-standard","hentry","category-docker","category-python","category-wallaroo"],"_links":{"self":[{"href":"https:\/\/qxf2.com\/blog\/wp-json\/wp\/v2\/posts\/8075","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/qxf2.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/qxf2.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/qxf2.com\/blog\/wp-json\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"https:\/\/qxf2.com\/blog\/wp-json\/wp\/v2\/comments?post=8075"}],"version-history":[{"count":12,"href":"https:\/\/qxf2.com\/blog\/wp-json\/wp\/v2\/posts\/8075\/revisions"}],"predecessor-version":[{"id":17220,"href":"https:\/\/qxf2.com\/blog\/wp-json\/wp\/v2\/posts\/8075\/revisions\/17220"}],"wp:attachment":[{"href":"https:\/\/qxf2.com\/blog\/wp-json\/wp\/v2\/media?parent=8075"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/qxf2.com\/blog\/wp-json\/wp\/v2\/categories?post=8075"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/qxf2.com\/blog\/wp-json\/wp\/v2\/tags?post=8075"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}