pytest: Cross browser, cross platform Selenium tests

We showed you how to run tests on BrowserStack using pytest in our previous post – pytest and Browserstack. We had an example test and ran it across one browser from the command line argument. What if we need to run it across different browser versions, platforms and platform versions?

Why this post?

pytest responds really well when we run it for one combination of browser, browser version, platform, and platform version. But to make it run across a combination of different browsers, browser versions, platforms and platform versions is a bit tricky. It is tricky because not all platforms support all browsers. E.g.: Safari on Windows does not make much sense. Further, your company may have promised different browser versions on different platforms as part of the minimum browser requirements. We are writing this post to explain a clean, maintainable way to run your Selenium tests against different browsers, browser versions, platforms and platform versions.

Overview

To get your tests running against a combination of browsers, browser versions and platforms, you need to do the following:
1. Parameterize the test in conftest.py
2. Generate the different browser/OS combinations you want
3. Generate the tests using pytest_generate_tests
4. Put it all together
5. Run the test

The rest of the post explains each of these steps in detail.

1. Parameterize the test in conftest.py

pytest allows easy parameterization of tests. To run a test against a list of browsers,

#contents of conftest.py
def pytest_generate_tests(metafunc):
    "test generator function to run tests across different parameters"
    if "browser" in metafunc.fixturenames:
        metafunc.parameterize("browser",metafunc.config.option.browser)

This function would parameterize the test if it finds the browser argument in the test function.
The corresponding changes that have to be made to the browser option are,

#Contents of conftest.py
parser.addoption("-B","--browser",
                 dest="browser",
                 action="append",
                 default=["firefox"],
                 help="Browser. Valid options are firefox, ie and chrome")

Now running the tests using the command py.test would run it across Firefox by default. py.test -B chrome command would append Chrome to the browser list for the test to run against. Hence the test would run against Firefox and Chrome.

2. Generate the different browser/OS combinations you want

Let us create a conf file to store the list of browsers, browser versions, platforms and platform versions for the test to run against. This conf file would hold a method to create a tuple of browser-appropriate browser versions and platform-appropriate platform versions.

#contents of the browser_platform_conf
"""
This is a contrived version of the browser_platfor_conf
We use a more detailed version in our code for clients
"""
browsers = ["chrome"]
os_list = ["windows","OS X"]
chrome_versions = ["49","50"]
windows_versions = ["7","8.1"]
os_x_versions = ["yosemite"]
def generate_configuration(browsers=browsers,firefox_versions=firefox_versions,chrome_versions=chrome_versions,os_list=os_list,windows_versions=windows_versions,os_x_versions=os_x_versions):
	"Generate test configuration"
	test_config = []
	for browser in browsers:
		if browser == "chrome":
			for chrome_version in chrome_versions:
				for os_name in os_list:
					if os_name == "windows":
						for windows_version in windows_versions:
							config = [browser,chrome_version,os_name,windows_version]
							test_config.append(tuple(config))
					if os_name == "OS X":
						for os_x_version in os_x_versions:
							config = [browser,chrome_version,os_name,os_x_version]
							test_config.append(tuple(config))
 
 
 
	return test_config
 
 
cross_browser_cross_platform_config = generate_configuration()

3. Generate the tests using pytest_generate_tests

pytest has a method called pytest_generate_tests. This method will ‘generate’ the same test for each of our browser/OS combinations. All we need to do, is to provide pytest_generate_tests with the different browser/OS combinations. The code snippet to do this, is:

#contents of conftest.py
from conf import browser_platform_conf
 
def pytest_generate_tests(metafunc):
    "test generator function to run tests across different parameters"
    if 'browser' in metafunc.fixturenames:
        if metafunc.config.getoption("-M").lower() == 'y':
            if metafunc.config.getoption("-B") == ["all"]:
                metafunc.parametrize("browser,browser_version,platform,os_version", 
                                    browser_platform_conf.cross_browser_cross_platform_config)

Refer this link to know more about pytest_generate_tests and the metafunc object.

4. Put it all together

Now to run the test, have the following files that look like the corresponding code below:
a. browser_platform_conf.py
b. conftest.py
c. Test file

4a. browser_platform_conf.py
The browser_platform_conf file to generate the test run configuration

"""
This is a contrived version of the browser_platform_conf
We use a more detailed version in our code for clients
"""
browsers = ["chrome"]
os_list = ["windows","OS X"]
chrome_versions = ["49","50"]
windows_versions = ["7","8.1"]
os_x_versions = ["yosemite"]
 
 
def generate_configuration(browsers=browsers,firefox_versions=firefox_versions,chrome_versions=chrome_versions,os_list=os_list,windows_versions=windows_versions,os_x_versions=os_x_versions):
 
	"Generate test configuration"
	test_config = []
	for browser in browsers:
		if browser == "chrome":
			for chrome_version in chrome_versions:
				for os_name in os_list:
					if os_name == "windows":
						for windows_version in windows_versions:
							config = [browser,chrome_version,os_name,windows_version]
							test_config.append(tuple(config))
					if os_name == "OS X":
						for os_x_version in os_x_versions:
							config = [browser,chrome_version,os_name,os_x_version]
							test_config.append(tuple(config))
 
 
 
	return test_config
 
 
cross_browser_cross_platform_config = generate_configuration()

4b. conftest.py
The conftest.py file to hold the fixtures and command line arguments and the pytest_generate_test to parametrize the test

import pytest
import os
import browser_platform_conf
 
 
@pytest.fixture
def browser():
    "pytest fixture for browser"
    return pytest.config.getoption("-B")
 
 
@pytest.fixture
def browserstack_flag():
    "pytest fixture for browserstack flag"
    return pytest.config.getoption("-M")
 
 
@pytest.fixture
def browser_version():
    "pytest fixture for browser version"
    return pytest.config.getoption("-V") 
 
 
@pytest.fixture
def platform():
    "pytest fixture for platform"
    return pytest.config.getoption("-P") 
 
 
@pytest.fixture
def os_version():
    "pytest fixture for os version"
    return pytest.config.getoption("-O") 
 
 
def pytest_generate_tests(metafunc):
    "test generator function to run tests across different parameters"
    if 'browser' in metafunc.fixturenames:
        if metafunc.config.getoption("-M").lower() == 'y':
            if metafunc.config.getoption("-B") == ["all"]:
                metafunc.parametrize("browser,browser_version,platform,os_version", 
                                    browser_platform_conf.cross_browser_cross_platform_config)
 
 
def pytest_addoption(parser):
    parser.addoption("-B","--browser",
                      dest="browser",
                      action="append",
                      default=[],
                      help="Browser. Valid options are firefox, ie and chrome")                      
    parser.addoption("-M","--browserstack_flag",
                      dest="browserstack_flag",
                      default="N",
                      help="Run the test in Browserstack: Y or N")
    parser.addoption("-O","--os_version",
                      dest="os_version",
                      action="append",
                      help="The operating system: xp, 7",
                      default=[])
    parser.addoption("-V","--ver",
                      dest="browser_version",
                      action="append",
                      help="The version of the browser: a whole number",
                      default=[])
    parser.addoption("-P","--platform",
                      dest="platform",
                      action="append",
                      help="The operating system: Windows 7, Linux",
                      default=[])

4c. Test file
Test file to test the selenium-tutorial-main from the previous post

#Contents of test file
"""
Qxf2 Services: Utility script to test example form on Browserstack
NOTE: This is a contrived example that was written up to make this blog post clear
We do not use this coding pattern at our clients
"""
 
from selenium import webdriver
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities
import sys,time
 
 
def test_example_form(browser,browserstack_flag,browser_version,platform,os_version,):
    "Test example form"
    #Create an instance of Driver_Factory
    driver = get_webdriver(browser,browser_version,platform,os_version)
    #Create variables to keep count of pass/fail
    pass_check_counter = 0
    total_checks = 0
    #Visit the tutorial page
    driver.get('http://qxf2.com/selenium-tutorial-main') 
    #Check 1: Is the page title correct?
    if(driver.title=="Qxf2 Services: Selenium training main"):
        print "\n"
        print ("Success: Title of the Qxf2 Tutorial page is correct")
        pass_check_counter += 1
    else:
        print ("Failed: Qxf2 Tutorial page Title is incorrect")
    total_checks += 1 
    #Check 2: Fill name, email and phone in the example form
    total_checks += 1
    name_field = driver.find_element_by_xpath("//input[@type='name']")
    name_field.send_keys('Shivahari')
    email_field = driver.find_element_by_xpath("//input[@type='email']")
    email_field.send_keys('[email protected]')
    phone_field = driver.find_element_by_xpath("//input[@type='phone']")
    phone_field.send_keys('9999999999')
    submit_button = driver.find_element_by_xpath("//button[@type='submit']") #Click on the Click me button
    try:
        submit_button.click()
        pass_check_counter += 1
    except Exception,e:
        print str(e)
    #Quit the browser window
    driver.quit() 
    #Assert if the pass and fail check counters are equal
    assert total_checks == pass_check_counter 
 
 
#NOTE: This is highly simplified code to make this post illustrative
#We do not use this code at clients
#We use Driver_Factory to return apporpriate drivers within our framework
def get_webdriver(browser,browser_version,platform,os_version):
    "Run the test in browser stack browser stack flag is 'Y'"
    USERNAME = user_name #We fetch values from a conf file in our framework we use on our clients
    PASSWORD = access_key
    if browser.lower() == 'firefox':
        desired_capabilities = DesiredCapabilities.FIREFOX
    if browser.lower() == 'chrome':
        desired_capabilities = DesiredCapabilities.CHROME
    desired_capabilities['os'] = platform
    desired_capabilities['os_version'] = os_version
    desired_capabilities['browser_version'] = browser_version
 
    return webdriver.Remote(command_executor='http://%s:%[email protected]:80/wd/hub'%(USERNAME,PASSWORD),
                            desired_capabilities=desired_capabilities)

5. Run the test

Run the test using the command py.test -v -M y -B all where,
-v is the verbose flag,
-M is the Browserstack flag
-B is the Browser option

test_run

And from now, all you need to do to add new browsers, browser versions, platforms is to edit the browser_platform_conf.py file. Happy testing!

If you liked what you read, know more about Qxf2.

Subscribe to our weekly Newsletter


View a sample


Related posts:

Default Thumbnailpytest and Browserstack Default ThumbnailPytest-xdist: Run tests in parallel Default ThumbnailPython+Selenium+pytest tutorial
Shivahari P

4 thoughts on “pytest: Cross browser, cross platform Selenium tests”

  2. Hi All,
    I am trying to run my selenium python test with BrowserStack .
    when i try to run with BrowserStack i get the below error :

    urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host=’hub-cloud.browserstack.com’, port=443): Max retries exceeded with url: /wd/hub/session (Caused by SSLError(SSLCertVerificationError(1, ‘[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1108)’)))

    This is my code snippet:

    def run_browserstack(self, os_name, os_version, browser, browser_version):
    “Run the test in browser stack when remote flag is ‘Y'”
    # Get the browser stack credentials from browser stack credentials file
    USERNAME = remote_credentials.USERNAME
    PASSWORD = remote_credentials.ACCESS_KEY
    desired_capabilities = DesiredCapabilities.CHROME
    desired_capabilities[‘os’] = os_name
    desired_capabilities[‘os_version’] = os_version
    desired_capabilities[‘browser_version’] = browser_version
    desired_capabilities[‘acceptSslCerts’] = True
    desired_capabilities[‘browserstack.local’] = True
    desired_capabilities[‘build’] = ‘version1’
    desired_capabilities[‘project’] = ‘loginpage’
    desired_capabilities[‘browserstack.debug’] = True

    driver = webdriver.Remote(
    command_executor=’https://{}:{}@hub-cloud.browserstack.com/wd/hub’.format(USERNAME, PASSWORD),
    desired_capabilities=desired_capabilities)
    return driver

    Reply

Leave a Reply

Your email address will not be published. Required fields are marked *