One of our clients recently introduced us to a really nice, Pythonic test runner called pytest. Thank you Vachan Wodeyar of Kahuna, Inc. for introducing and helping us get started with pytest!
 
Our GUI automation framework lacked a good test runner. So it was tough to run all our tests with just one command. It also made reporting results to a CI server somewhat limited. Sigh us! We knew we lacked a good test runner. It was just that the standard options (nose, unittest, etc.) did not fit our thinking. But with pytest, we seem to have found a really good test runner that fits us. So if you are like us, unhappy about the current state of test runners in Python, do consider using pytest.

Why this post?

The documentation for most test runners, are written for people who had the time and resources to start with a test runner. However, integrating a test runner with an existing test suite is a genuine problem that many testers face. We try to address this problem in this post. We’ll show you how to modify your existing tests to use pytest.

In this post, we’ll assume the problem before you is this:
1. you already have a set of GUI automation scripts written in Python
2. you want to add a test runner on top of your existing tests
3. your tests do not follow the unit test pattern of asserting every single check

When do I need a test runner?

As testers at startups, we rarely get an opportunity to focus on only automation. We split up our time between testing new features and writing automation for the most important features of the application. Even when we write automation, our first tests are of the ‘super hero’ variety – the ones that catch only the ‘super villains’. Why do we do that? Because we have limited time and a broadstack test exercising multiple workflows is an efficient use of time and money. At this stage, it is easy enough to get by without a test runner. Just make the tests flexible with command line options. Provide a way for your developers to run them after every commit. Once your startup gets financially healthy, your test suite will begin to grow – one test at a time. At some point, you will have a hard time keeping track of every test added. You will also have a hard time communicating what tests your development team should run. Further, the results you post on your CI server will start getting very hard to read and maintain. This is about the time you should start looking for a test runner.

An existing GUI automation test

Let us pretend that we have an existing automated test that fills out a form on this page. The test will fill the name, phone number, email and then click submit. Let us also try running this test against Firefox or the Chrome based on a command line option. Let us also assume we have functional style test.

The initial test script, say fill_example_form.py would look something like this-

"""
Qxf2 Services: Utility script to test example form
NOTE: This is a contrived example that was written up to make this blog post clear
We do not use this coding pattern at our clients
"""
 
from selenium import webdriver
import sys,time
 
def fill_example_form(browser):
	"Test example form"
 
	#Create an instance of WebDriver
	if browser.lower() == 'firefox':
            driver = webdriver.Firefox()
	elif browser.lower() == 'chrome':
            driver = webdriver.Chrome()
 
	#Create variables to keep count of pass/fail
	pass_check_counter = 0
	total_checks = 0
 
        #Visit the tutorial page
	driver.get('http://qxf2.com/selenium-tutorial-main') 
        #Check 1: Is the page title correct?
        if(driver.title=="Qxf2 Services: Selenium training main"):
            print ("Success: Title of the Qxf2 Tutorial page is correct")
            pass_check_counter += 1
        else:
            print ("Failed: Qxf2 Tutorial page Title is incorrect")
	total_checks += 1
 
	#Fill name, email and phone in the example form
	name_field = driver.find_element_by_xpath("//input[@type='name']")
        name_field.send_keys('Shivahari')
	email_field = driver.find_element_by_xpath("//input[@type='email']")
        email_field.send_keys('[email protected]')
	phone_field = driver.find_element_by_xpath("//input[@type='phone']")
        phone_field.send_keys('9999999999')
	submit_button = driver.find_element_by_xpath("//button[@type='submit']") #Click on the Click me button
        submit_button.click()
        time.sleep(5)
        #Check 2: Is the page title correct?
        if(driver.title=="Qxf2 Services: Selenium training redirect"):
            print ("Success: The example form was submitted")
            pass_check_counter += 1
        else:
            print ("Failed: The example form was not submitted. Automation is not on the redirect page")
	total_checks += 1
	#Quit the browser window
	driver.quit() 
 
	#Assert if the pass and fail check counters are equal
	assert total_checks == pass_check_counter 
 
#---START OF SCRIPT
if __name__=='__main__':
    browser = sys.argv[1] #Note:using sys.argv to keep this example short. We use OptionParser with all our clients 
    fill_example_form(browser)

Sure, this is a contrived example, but we feel like its simplicity allows us to illustrate our point clearly.

Setup

To install the pytest module use:

pip install -U pytest

To check the pytest version installed use:

py.test --version

A brief detour: how pytest gathers tests

Before you begin making changes to your existing tests, it is worth understanding the convention pytest uses to magically discover your tests and execute them. pytest runs all the files in the current directory and it’s sub-directories, if the files follow these conventions:
a. The test files start with test_*.py or end with *_test.py
b. Test classes prefixed with Test that have no __init__ method
c. Test functions or methods prefixed with test_

Modify the existing test

Modifying tests to run with pytest is very easy. Just make sure:
1. Your tests follow the naming conventions in the previous section
2. If you don’t have assert statements, add one at the end of each test method
3. Move your command line options to a separate conftest.py file
4. Add empty __init__.py file in the sub-directories that contain tests in them

Step 1. Modify the naming patterns

In the previous test,
a. change the name of the test file to test_example_form.py
b. change the method name from fill_example_form to test_example_form

Step 2. Add an assert statement

pytest uses the assert statement to figure out if a test Passed or Failed. If no assert statement is used then the test result shows Passed irrespective of the test outcome

Step 3. Move your command line options to a separate conftest.py file

To support command line dependencies for a test, create a conftest.py file in the root directory of your code repository. Add the command line options you use to conftest.py. Then create fixtures to fetch the value from the config file every time the test is run. Your conftest.py should look something like this:

#Contents of conftest.py
import pytest
 
#Command line options:
#Example of allowing pytest to accept a command line option
def pytest_addoption(parser):
    parser.addoption("-B","--browser",
                      dest="browser",
                      default="firefox",
                      help="Browser. Valid options are firefox or chrome")
 
#Test arguments:
#Example of populating the argument 'browser' for a test 
@pytest.fixture
def browser():
    "pytest fixture for browser"
    return pytest.config.getoption("-B")

In our example, our test accepts one command line option: the browser. So let pytest know it should accept a command line option for browser too. Our test method took one argument called browser. So create a fixture called browser() that to read the value for browser.

Note: There is a lot more you could do with pytest fixtures. A description of the uses of fixtures would be detailed in future posts.

4. Add empty __init__.py file in the sub-directories that contain tests in them

Add empty files called __init__.py to every directory that you want pytest to look for tests in. This is actually optional depending upon your folder structure and what arguments you plan on running pytest with. But if you are starting off with pytest, just perform this step. You can remove the empty __init__.py files once you get a hang of pytest.

Putting it all together

For the sake of completeness, this is how our modified test, now named test_example_form.py looks like.

"""
Qxf2 Services: Utility script to test example form
NOTE: This is a contrived example that was written up to make this blog post clear
We do not use this coding pattern at our clients
"""
 
from selenium import webdriver
import sys,time
 
def test_example_form(browser):
    "Test example form"
 
    #Create an instance of WebDriver
    if browser.lower() == 'firefox':
            driver = webdriver.Firefox()
    elif browser.lower() == 'chrome':
            driver = webdriver.Chrome()
 
    #Create variables to keep count of pass/fail
    pass_check_counter = 0
    total_checks = 0
 
    #Visit the tutorial page
    driver.get('http://qxf2.com/selenium-tutorial-main') 
    #Check 1: Is the page title correct?
    if(driver.title=="Qxf2 Services: Selenium training main"):
        print ("Success: Title of the Qxf2 Tutorial page is correct")
        pass_check_counter += 1
    else:
        print ("Failed: Qxf2 Tutorial page Title is incorrect")
    total_checks += 1
 
    #Fill name, email and phone in the example form
    name_field = driver.find_element_by_xpath("//input[@type='name']")
    name_field.send_keys('Shivahari')
    email_field = driver.find_element_by_xpath("//input[@type='email']")
    email_field.send_keys('[email protected]')
    phone_field = driver.find_element_by_xpath("//input[@type='phone']")
    phone_field.send_keys('9999999999')
    submit_button = driver.find_element_by_xpath("//button[@type='submit']") #Click on the Click me button
    submit_button.click()
    time.sleep(5)
    #Check 2: Is the page title correct?
    if(driver.title=="Qxf2 Services: Selenium training redirect"):
        print ("Success: The example form was submitted")
        pass_check_counter += 1
    else:
        print ("Failed: The example form was not submitted. Automation is not on the redirect page")
    total_checks += 1
    #Quit the browser window
    driver.quit() 
 
    #Assert if the pass and fail check counters are equal
    assert total_checks == pass_check_counter 
 
#---START OF SCRIPT
if __name__=='__main__':
    browser = sys.argv[1] #Note:using sys.argv to keep this example short. We use OptionParser with all our clients 
    text_example_form(browser)

How to run tests:

Pull up a terminal, go to the root directory of your code repository and run the py.test command. The simplest way to run all the tests use the command py.test. We’ll show you an example of running the test in verbose mode: py.test -v

There are many more default options that can be used in pytest. Use py.test -h to list the default options, custom options,ini-options and environment variables.

That’s it! Your tests have been modified to run with pytest. We plan on covering a lot more ground with pytest – but if you are itching to pick up something specific, do let us know in the comments below.

If you liked this article, learn more about Qxf2’s testing services for startups.

Related posts:

pytest: Cross browser, cross platform Selenium tests pytest and Browserstack Post pytest test results on Slack