Problem: How do you run tests in parallel with pytest?
We have begun using pytest as our test runner at most of our clients. At one client, our tests are triggered by CircleCI and run on BrowserStack. Our GUI automation suite grew in size and was starting to take a long time to run against a vast combination of OSes and browser versions. Luckily, our client purchased 5 BrowserStack licenses for us. We wanted to run tests in parallel on the 5 channel BrowserStack from CircleCI using pytest. We searched on Google, but we didn’t get much information about running pytest in parallel. So this post is to help testers facing a similar issue. We will show you how to use pytest-xdist to run pytests in parallel.
Setup:
To run pytests in parallel, we need to install pytest-xdist plugin. To install x-dist plugin use:
pip install pytest-xdist |
Command to run pytest in parallel:
The command to tell pytest to run your tests is parallel depends on the OS.
A)Windows:
To run tests in parallel on Windows, use:
py.test –n NUM |
Here NUM = Number of parallel tests you want running at one time
In our case, we had 5 BrowserStack channels. So our command ended up being:
py.test –n 5 |
B)Linux:
Our CircleCI container runs Ubuntu. To run tests in parallel from on Linux, use:
py.test -d --tx NUM*popen//python=python2.7 |
Here NUM = Number of parallel tests you want running at one time
In our case, our command ended up being:
py.test -d --tx 5*popen//python=python2.7 |
We liked pytest-xdist. It helped us distribute and run our tests in parallel in a straightforward way. We hope this helps you too.
This article was made possible by Qxf2’s commitment to sharing our work with the larger testing community. Qxf2 specializes in QA for early-stage products, understanding the unique demands startups face in testing. Our offerings include automation, consulting, and on-demand expertise tailored to startup needs. Explore our quality assurance solutions for startups that deliver real value right from the start.
I love technology and learning new things. I explore both hardware and software. I am passionate about robotics and embedded systems which motivate me to develop my software and hardware skills. I have good knowledge of Python, Selenium, Arduino, C and hardware design. I have developed several robots and participated in robotics competitions. I am constantly exploring new test ideas and test tools for software and hardware. At Qxf2, I am working on developing hardware tools for automated tests ala Tapster. Incidentally, I created Qxf2’s first robot. Besides testing, I like playing cricket, badminton and developing embedded gadget for fun.
The -n argument works the same under Linux as it does under Windows. Is there a particular reason to take the longer form?
Hi,
Thanks for pointing it out.We tried running tests using -n on Linux using Circle CI and it did not work hence the long form.
Confirmed that the -n argument works in Linux.
Hi,
Thanks for responding.
What distribution of Linux did you try it against?
How can we run all the test_.py files from the python script itself. I want to run all of my test_.py files by calling a function with a python script using tkinter . Can you kindly help advise.
Hi Jalal,
To run all the test_.py files from python script, you need to add following lines in python script
call_pytest_file.py
import subprocessprint subprocess.check_output(['pytest'])
if you pytest command which you want to use contents spaces, you need follow below fashion
#subprocess.check_output([‘ls’,’-l’])
Hope it will help you
thanks
While using xdist Im not able to call the pytest hook makereport
Can you please refer the following link, it may help you to solve your issue.
https://github.com/pytest-dev/pytest-xdist/issues/79
Can xdist run test parallel on multiple node instead of multiple browser?
Hi Can you refer to this link https://github.com/pytest-dev/pytest-xdist#multi-platform and pay attention to ‘–dist=loadfile’ & ‘–dist=loadscop’ params there?
If I have multiples tests that each take different command line arguments how do I use xdist to run them in parallel.
Example:
testsuite/
test1.yaml
test2.yaml
test1.py
test2.py
If I were to run these tests serially the commands would look like
python -m test1.py –config=test1.yaml
python -m test2.py –config=test2.yaml
You can run the tests in parallel like below:
pytest -n auto -m parallel test1.py –config=test1.yaml && pytest -n auto -m parallel test2.py –config=test2.yaml– html=report.html
Check this too:
https://github.com/pytest-dev/pytest-xdist/issues/325