Selenium Grid enables testers to parallelize their testing pipeline. It can be used to automate the process of manually setting up a distributed test environment where you can expose the code to these environments. This helps speed up the cycle of Continuous Integration. This can be sped up even further by using the capabilities of Docker Swarm. This article will show you some scripts which we built to help to quickly create and configure a cluster rapidly.<\/p>\n
Prerequisites<\/h3>\n
This post assumes prior knowledge of Docker<\/a> and some experience working with Docker. You should also be familiar with the following AWS EC2 services.<\/a><\/p>\n By the end of this tutorial, you will be able to run some sample Selenium tests parallelly across AWS machine setup having Chrome and Firefox drivers. To achieve this you need to <\/p>\n 1) Automate the provisioning of resources on AWS Note:- This article will walk you through setting up a Grid on AWS using 3 machines (1 manager, 2 worker nodes). Before going into details lets have a brief about Selenium grid<\/a> and Docker Swarm.<\/a><\/p>\n Selenium-Grid<\/a> allows you to run your tests on different machines against different browsers in parallel. The entry point of Selenium Grid is a Selenium Hub. Our test cases will hit the hub and spin up whatever browser is available within your Grid using the DesiredCapabilities function of Selenium. Next elements are nodes, which are machines that are registered to the hub which can execute the test cases. To run multiple tests in parallel, a grid is a must. Docker Swarm<\/a> is a tool used to cluster and orchestrate Docker containers. There are two types of nodes: manager nodes (to define services), and worker nodes (instructed by manager nodes based on the service definition). You submit a service definition to a manager node. The service definition consists of one or more tasks and how many replicas of that service you want to run on the cluster. <\/p>\n We wrote two scripts, swarm.sh<\/em><\/strong> and swarmdown.sh<\/em><\/strong>. The swarmdown.sh <\/em><\/strong> will<\/p>\n Here is the complete swarmup.sh script. We will have a look at each section in detail below.<\/p>\n swarmup.sh<\/em><\/strong><\/p>\n We have created 3 EC2 instances of t2.micro instance type using ubuntu image(By default, the Amazon EC2 driver will use a daily image of Ubuntu 15.10 LTS so that latest version of Docker can be installed)<\/p>\n The first step in the script starts with spinning 3 EC2 instances. Before continuing to creating the instances let’s first understand about credentials configuration and ports that we need to configure in the security group.<\/p>\n Credentials Configuration:<\/strong> First step is to configure credentials. This can either be specified from command line flags or put in an environment file. We used environment variables to set the credentials. You can set the environment variables using the You can find more details about configuring credentials here<\/a><\/p>\n Port Configuration:<\/strong> Docker Swarm requires few ports to be open for it to work. <\/p>\n Below is the command to spin up 3 EC2 instances<\/p>\n Docker-machine has an EC2 driver for creating a Docker node out of AWS. Docker node in this context means an AWS VM instance with Docker pre-installed. With default options, The below commands allows you to ssh into each instances and update and upgrade os.<\/p>\n In our case, the swarm will be composed of, 1 node with the manager role and 2 nodes with the worker role. So we would be creating swarm cluster by making “swarmmode-1” node as master and “swarmmode-2” node as worker1 and “swarmmode-3” as worker2. For achieving this, we can initialize the swarm cluster on the manager node using “ The above code will add the first node as manager node and remaining two nodes to the swarm as workers. When these commands are executed you should see something like this in the console. You can run the script with below command<\/p>\n Next, create a Docker compose file and place in the same directory where your shell scripts are located. Our docker-compose.yml looks like this. Docker compose lets you deploy Selenium Grid in multiple containers. The .yml file describes the containers and how they interact with each other. We are using standard SeleniumHQ images with Chrome and Firefox browsers already installed for hub and nodes. In this article, we will be using below docker images:<\/p>\n selenium\/hub<\/strong> : Image for running a Grid Hub. It will expose port 4444 on the AWS instance so we can connect to the Selenium Grid. Using this .yml file, we will be configuring the services. In this case, we have a hub service with two node services(Chrome and Firefox). When you deploy this .yml file on the Docker Swarm, all the services will be running on nodes from this configuration.<\/p>\n docker-compose.yml<\/strong><\/em><\/p>\n Before deploying the Selenium Grid on Docker Swarm, let’s look into small details which need attention.<\/p>\n Placement constraints<\/strong>: [node.role == worker] allows placing the workload on the worker node instead of the manager nodes. In case you want to run the hub on the manager node use [node.role == manager] as the value instead. But best practice is to keep manager nodes free from CPU and\/or memory-intensive tasks.<\/p>\n Entrypoint<\/strong>: entry_point.sh gets the info to register the node to the hub. Hub also needs an address of the node to poll its status. We can use ‘SE_OPTS’ within the entry_point.sh script so nodes running on different hosts will be able to successfully link back to the hub. <\/p>\n Port mapping<\/strong>: We have exposed the ports as 4444:4444 which basically means one can connect to the grid on port 4444 on any node in the Swarm network. <\/p>\n The below command deploys the docker stack into the manager node:<\/p>\n Here is the Grid console<\/p>\n Now ssh into the swarmmode-1 instance and run below command:<\/p>\n And review the stack:<\/p>\n For testing the grid setup, we wrote couple of test scripts. Both the scripts loads qxf2.com and prints the title of the page. In your local machine, from the directory where you have these scripts, run below command<\/p>\n test_selenium_grid_chrome.py<\/em><\/p>\n test_selenium_grid_firefox.py<\/em><\/p>\n Review the service:<\/p>\n If you want to scale the grid nodes use the This command would scale selenium_test_chrome to 4. You can even apply Auto Scaling for adding new worker nodes based on the load.<\/p>\n Since the resources used are all virtual, launching them as part of an experiment or to solve some short-term problem will often make lots of sense. When the work’s all done, the resource can be shut down. We wrote a script to achieve this. From the directory where you saved this script, execute below command to run this script<\/p>\n To test the above scripts, from your local machine you can run below.<\/p>\n The main objective of this article is to automatically provision the instances before test runs and then de-provision them after. You can configure to run tests in parallel using pytest<\/a> in Jenkins <\/a>(or some other CI tool) so that they are part of the continuous integration process. <\/p>\n So now you can have your own testing lab on the cloud. Happy testing!<\/p>\n 1) Distributed testing with selenium grid<\/a> Maintaining infrastructure for automated Selenium cross-browser tests is time-consuming. The cloud testing platforms like BrowserStack and Saucelabs help you. But in some cases, you want to have your own cloud testing environment. This is usually time-consuming and involves setting up and using Selenium Grid. This post helps testers to automate the setup process for cloud-based testing. We hope this post […]<\/p>\n","protected":false},"author":16,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[38,177,52,175,30,176],"tags":[],"class_list":["post-10018","post","type-post","status-publish","format-standard","hentry","category-automation","category-aws","category-continuous-integration","category-docker-swarm","category-selenium","category-selenium-grid"],"_links":{"self":[{"href":"https:\/\/qxf2.com\/blog\/wp-json\/wp\/v2\/posts\/10018","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/qxf2.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/qxf2.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/qxf2.com\/blog\/wp-json\/wp\/v2\/users\/16"}],"replies":[{"embeddable":true,"href":"https:\/\/qxf2.com\/blog\/wp-json\/wp\/v2\/comments?post=10018"}],"version-history":[{"count":50,"href":"https:\/\/qxf2.com\/blog\/wp-json\/wp\/v2\/posts\/10018\/revisions"}],"predecessor-version":[{"id":12301,"href":"https:\/\/qxf2.com\/blog\/wp-json\/wp\/v2\/posts\/10018\/revisions\/12301"}],"wp:attachment":[{"href":"https:\/\/qxf2.com\/blog\/wp-json\/wp\/v2\/media?parent=10018"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/qxf2.com\/blog\/wp-json\/wp\/v2\/categories?post=10018"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/qxf2.com\/blog\/wp-json\/wp\/v2\/tags?post=10018"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}Objective<\/h3>\n
\n2) Set up the swarm manager and workers
\n3) Create the Selenium Grid
\n4) Deploy the Selenium Grid to AWS via Docker Compose
\n5) Run automated tests on Selenium Grid
\n6) Automate de-provisioning of resources on AWS<\/p>\nWhat is Selenium Grid?<\/h3>\n
\n<\/hr>\nWhat is Docker Swarm?<\/h3>\n
About our automation scripts<\/h3>\n
\nThe swarm.sh<\/em><\/strong> will<\/p>\n\r\n#!\/bin\/bash\r\n\r\necho \"Spinning up three aws instances...\"\r\nfor i in 1 2 3 ; do\r\n\tdocker-machine create \\\r\n\t\t--driver amazonec2 \\\r\n\t\t--amazonec2-open-port 2377 \\\r\n\t\t--amazonec2-open-port 7946 \\\r\n\t\t--amazonec2-open-port 4789 \\\r\n\t\t--amazonec2-open-port 7946\/udp \\\r\n\t\t--amazonec2-open-port 4789\/udp \\\r\n\t\t--amazonec2-open-port 8080 \\\r\n\t\t--amazonec2-open-port 80 \\\r\n\t\t--amazonec2-open-port 22 \\\r\n\t\t--amazonec2-open-port 4444 \\\r\n\t\tswarmmode-$i\r\ndone\t\t\r\n\r\n#update os\r\nfor i in 1 2 3; do\r\n\techo \"updating swarmmode-$i\"\r\n docker-machine ssh swarmmode-$i 'apt-get update && apt-get upgrade -y && reboot'\r\ndone\r\n\r\nsleep 10\r\n\r\necho \"Initializing Swarm mode...\"\r\nfor i in 1 2 3; do\r\n\tif [ \"$i\" == \"1\" ]; then\r\n manager_ip=$(docker-machine ip swarmmode-$i)\r\n\t\teval $(docker-machine env swarmmode-$i) && \\\r\n docker swarm init --advertise-addr \"$manager_ip\"\r\n worker_token=$(docker swarm join-token worker -q)\r\n\telse \r\n\t\teval $(docker-machine env swarmmode-$i) && \\\r\n docker swarm join --token \"$worker_token\" \"$manager_ip:2377\"\r\n fi\r\ndone\r\n\r\necho \"Deploying Selenium Grid to http:\/\/$(docker-machine ip swarmmode-1):4444...\"\r\n\r\neval $(docker-machine env swarmmode-1)\r\ndocker stack deploy --compose-file=docker-compose.yml selenium_test\r\n<\/pre>\n
Provisioning resources to EC2<\/h3>\n
export<\/code>command as shown below<\/p>\n\r\n$ export AWS_ACCESS_KEY_ID=MY-ACCESS-ID\r\n$ export AWS_SECRET_ACCESS_KEY=MY-SECRET-KEY<\/pre>\n
\r\nfor i in 1 2 3 ; do\r\n\tdocker-machine create \\\r\n\t\t--driver amazonec2 \\\t\t\t\r\n\t\t--amazonec2-open-port 2377 \\\r\n\t\t--amazonec2-open-port 7946 \\\r\n\t\t--amazonec2-open-port 4789 \\\r\n\t\t--amazonec2-open-port 7946\/udp \\\r\n\t\t--amazonec2-open-port 4789\/udp \\\r\n\t\t--amazonec2-open-port 8080 \\\r\n\t\t--amazonec2-open-port 80 \\\r\n\t\t--amazonec2-open-port 22 \\\r\n\t\t--amazonec2-open-port 4444 \\\r\n\t swarmmode-$i\r\ndone\r\n<\/pre>\n
docker-machine<\/code> picks up a t2.micro EC2 node with Ubuntu 15.10 OS and installs latest Docker engine on the AWS instance. docker-machine<\/code> also takes care of configuring appropriate certificates which allows us to access the AWS VM securely. You can refer to this link<\/a> for more details.<\/p>\nSsh into each instance and update the os<\/h3>\n
\r\nfor i in 1 2 3; do\r\n\techo \"updating swarmmode-$i\"\r\n docker-machine ssh swarmmode-$i 'apt-get update && apt-get upgrade -y && reboot'\r\ndone\r\n<\/pre>\n
Setting up the swarm manager and workers<\/h3>\n
docker swarm init<\/code>“. The command docker swarm join-token worker -q<\/code> gives the worker_token which is needed to add node workers to the swarm. <\/p>\n\r\nfor i in 1 2 3; do\r\n\tif [ \"$i\" == \"1\" ]; then\r\n manager_ip=$(docker-machine ip swarmmode-$i)\r\n\t\teval $(docker-machine env swarmmode-$i) && \\\r\n docker swarm init --advertise-addr \"$manager_ip\"\r\n worker_token=$(docker swarm join-token worker -q)\r\n\telse \r\n\t\teval $(docker-machine env swarmmode-$i) && \\\r\n docker swarm join --token \"$worker_token\" \"$manager_ip:2377\"\r\n fi\r\ndone\r\n<\/pre>\n
\r\n$ sh swarmup.sh\r\n<\/pre>\n
\r\nSwarm initialized: current node (deyp099wnn94lgauxp9ljil83) is now a manager.\r\n\r\nTo add a worker to this swarm, run the following command:\r\n\r\n docker swarm join --token SWMTKN-1-5rm2sib935txv5k13j6leaqsfuuttalktt7jv4s55249izjf54-8ia31tagc4sbehqeqiqst4jfz 172.30.0.170:2377\r\n\r\nTo add a manager to this swarm, run 'docker swarm join-token manager' and follow the instructions.\r\n\r\nThis node joined a swarm as a worker.\r\nThis node joined a swarm as a worker.\r\n\r\n<\/pre>\n
Creating the Selenium Grid<\/h3>\n
\nselenium\/node-chrome<\/strong> : Grid Node with Chrome installed.
\nselenium\/node-firefox<\/strong> : Grid Node with Firefox installed.<\/p>\n\r\n\r\nversion: \"3.5\"\r\nnetworks:\r\n main:\r\n driver: overlay\r\nservices:\r\n hub:\r\n image: selenium\/hub\r\n ports:\r\n - \"4444:4444\"\r\n networks:\r\n - main\r\n deploy:\r\n mode: replicated\r\n replicas: 1\r\n labels:\r\n selenium.grid.type: \"hub\"\r\n selenium.grid.hub: \"true\"\r\n restart_policy:\r\n condition: none\r\n placement:\r\n constraints: [node.role == manager ]\r\n chrome:\r\n image: selenium\/node-chrome\r\n entrypoint: >\r\n bash -c '\r\n export IP_ADDRESS=$$(ip addr show eth0 | grep \"inet\\b\" | awk '\"'\"'{print $$2}'\"'\"' | awk -F\/ '\"'\"'{print $$1}'\"'\"' | head -1) &&\r\n SE_OPTS=\"-host $$IP_ADDRESS\" \/opt\/bin\/entry_point.sh'\r\n volumes:\r\n - \/dev\/urandom:\/dev\/random\r\n - \/dev\/shm:\/dev\/shm\r\n depends_on:\r\n - hub\r\n environment:\r\n HUB_PORT_4444_TCP_ADDR: hub\r\n HUB_PORT_4444_TCP_PORT: 4444\r\n NODE_MAX_SESSION: 1\r\n networks:\r\n - main\r\n deploy:\r\n mode: replicated\r\n replicas: 1\r\n labels:\r\n selenium.grid.type: \"node\"\r\n selenium.grid.node: \"true\"\r\n restart_policy:\r\n condition: none\r\n placement:\r\n constraints: [node.role == worker]\r\n firefox:\r\n image: selenium\/node-firefox\r\n entrypoint: >\r\n bash -c '\r\n export IP_ADDRESS=$$HOSTNAME &&\r\n SE_OPTS=\"-host $$IP_ADDRESS\" \/opt\/bin\/entry_point.sh'\r\n volumes:\r\n - \/dev\/shm:\/dev\/shm\r\n - \/dev\/urandom:\/dev\/random\r\n depends_on:\r\n - hub\r\n environment:\r\n HUB_PORT_4444_TCP_ADDR: hub\r\n HUB_PORT_4444_TCP_PORT: 4444\r\n NODE_MAX_SESSION: 1\r\n networks:\r\n - main\r\n deploy:\r\n mode: replicated\r\n replicas: 1\r\n labels:\r\n selenium.grid.type: \"node\"\r\n selenium.grid.node: \"true\"\r\n restart_policy:\r\n condition: none\r\n placement:\r\n constraints: [node.role == worker]\r\n<\/pre>\nDeploying the Selenium Grid to AWS via Docker Compose<\/h3>\n
\r\n\r\n$ docker stack deploy --compose-file docker-compose.yml selenium_test\r\nDeploying Selenium Grid to http:\/\/100.24.107.127:4444...\r\nCreating network selenium_test_main\r\nCreating service selenium_test_hub\r\nCreating service selenium_test_chrome\r\nCreating service selenium_test_firefox\r\n<\/pre>\n
![\"\"](\"https:\/\/qxf2.com\/blog\/wp-content\/uploads\/2018\/10\/seleniumgrid-1024x221.jpg\")
<\/a>\r\n$docker-machine ssh swarmmode-1<\/pre>\n
\r\n$docker stack ps selenium_test<\/pre>\n
![\"\"](\"https:\/\/qxf2.com\/blog\/wp-content\/uploads\/2018\/10\/stack_ps-1024x57.jpg\")
<\/a>Run tests on Selenium Grid<\/h3>\n
\r\n$ pytest -k test\r\n<\/pre>\n
![\"\"](\"https:\/\/qxf2.com\/blog\/wp-content\/uploads\/2018\/11\/pytest-grid.jpg\")
<\/a>\r\n'''Test for launching qxf2.com in AWS machine through grid'''\r\n\r\nfrom selenium import webdriver\r\nfrom selenium.webdriver.common.desired_capabilities import DesiredCapabilities\r\n\r\ndef test_chrome():\r\n #Set desired capabilities\r\n desired_capabilities = DesiredCapabilities.CHROME\r\n desired_capabilities['platform'] = 'LINUX'\r\n\r\n #Using Remote connection to connect to aws grid - Used manager ip\r\n url = \"http:\/\/100.24.107.127:4444\/wd\/hub\"\r\n driver = webdriver.Remote(url, desired_capabilities)\r\n\r\n #Checking the driver session\r\n print driver\r\n\r\n #Launch qxf2 and print title\r\n driver.get(\"http:\/\/www.qxf2.com\")\r\n\r\n print driver.title\r\n driver.quit()\r\n<\/pre>\n
\r\n'''Test for launching qxf2.com in AWS machine through grid'''\r\n\r\nfrom selenium import webdriver\r\nfrom selenium.webdriver.common.desired_capabilities import DesiredCapabilities\r\n\r\ndef test_firefox():\r\n #Set desired capabilities\r\n desired_capabilities = DesiredCapabilities.FIREFOX\r\n desired_capabilities['platform'] = 'LINUX'\r\n\r\n #Using Remote connection to connect to aws grid - Used manager ip\r\n url = \"http:\/\/100.24.107.127:4444\/wd\/hub\"\r\n driver = webdriver.Remote(url, desired_capabilities)\r\n\r\n #Checking the driver session\r\n print driver\r\n\r\n #Launch qxf2 and print title\r\n driver.get(\"http:\/\/www.qxf2.com\")\r\n\r\n print driver.title\r\n driver.quit()\r\n<\/pre>\n
![\"\"](\"https:\/\/qxf2.com\/blog\/wp-content\/uploads\/2018\/10\/docker_service-1024x62.jpg\")
<\/a>scale<\/code>command on the Swarm manager: <\/p>\n\r\n$ docker service scale selenium_test_chrome=4\r\n<\/pre>\n
Automate de-provisioning of resources on AWS<\/h3>\n
\nswarmdown.sh<\/em><\/p>\n\r\n#!\/bin\/bash\r\n\r\ndocker-machine rm swarmmode-1 swarmmode-2 swarmmode-3 -y\r\n\r\n<\/pre>\n
\r\nsh swarmdown.sh\r\n<\/pre>\n
\r\n$ sh swarmup.sh\r\n$ pytest -k test\r\n$ sh swarmdown.sh\r\n\r\n<\/pre>\n
References:<\/h3>\n
\n2) Running selenium grid with docker<\/a>
\n3) When to use Docker-Compose and when to use Docker-Swarm<\/a>
\n4) Github – Docker swarm scripts<\/a><\/p>\n