ReportPortal integration with pytest and pytest markers

Problem: If QA wants to make a difference with their testing and influence decision-making, we should convey our test results clearly. Hence test reporting becomes an important part of testing.
 
There are several test reporting tools available and recently many AI based test reporting tools are getting popular. Integrating with these reporting tools is useful. In this post, we will show you how to integrate a Python test which uses pytest as the test runner to integrate with a popular test reporting tool ReportPortal. ReportPortal helps to acquire, aggregate and analyze test reports to ascertain release health better. It also shows us trends over time.
 
Thank you Lea Anthony of Secure Code Warrior for introducing us to ReportPortal!


Why this post?

One of our clients was planning on using ReportPortal as the test reporting tool. The plain, vanilla integration of ReportPortal with pytest (which is the default test runner we use at Qxf2 Services) is straightforward and covered well by ReportPortal’s docs. The plugin for reporting test results of Pytest to the ReportPortal needs the pytest.ini to be updated at a root level. This was a problem for us on two counts:
a) we wanted to be able to have different ReportPortal projects (called “launch name” in ReportPortal) for different pytest markers. We know the marker only after the pytest.ini has been initialized. So the launch name used couldn’t be conditionally set in pytest.ini
b) we like the config parameters to be set somewhere other than declaring it directly in the pytest.ini file


How to integrate ReportPortal with pytest markers

To illustrate how to integrate ReportPortal with pytest markers, we will be running two tests which fall under API and GUI test groups. When we use the marker as gui_test, the GUI test should run and report to ReportPortal with a launch name as gui_test. When we use the marker as api_test, the API test should run and report to ReportPortal with a launch name as api_test.

Below are the steps needed to achieve this
1. Set up ReportPortal
2. Create two tests with markers
3. Set up pytest plugin for ReportPortal
4. Create a conftest
5. Run the test

1. Set up ReportPortal
You can go about downloading and setting up ReportPortal as mentioned or try their Demo server. I have used the demo server for this blog. Navigate to profile link and make a note of UUID, endpoint and project details.

ReportPortal_Profile
2. Create two tests with markers

For the demo, we will create dummy tests with just the function names. We will add a couple of markers for gui and api tests.
Note: With pytest markers, you can restrict a test run to only run tests only with specific markers

import pytest
 
@pytest.mark.gui_test()
def test_gui():
    print ("GUI Test")
    pass
 
@pytest.mark.api_test()
def test_api():
    print ("API Test")
    pass

3. Set up pytest plugin for ReportPortal
The agent-python-pytest plugin allows you to integrate pytest with ReportPortal. To install the plugin you can run

pip install pytest-reportportal

The below example is the default way suggested for integrating with ReportPortal. You can skip this if you want to use a more flexible pytest hook implementation instead(discussed next).

Add the ReportPortal details like UUID, endpoint and project detail which you noted in step 2 and use it in pytest.ini file. Example:

[pytest]
rp_uuid = fb586627-32be-47dd-93c1-678873458a5f
rp_endpoint = http://192.168.1.10:8080
rp_project = user_personal
rp_launch = AnyLaunchName

Note: The UUID and other details mentioned above are for my Demo server and needs to be updated as per your ReportPortal details.

4. Creating a conftest
However, instead of the above example, we can to create a conftest file. If you use pytest regularly, you probably already have one! If not, create a conftest.py file where we can add the ini details conditionally which ReportPortal requires

a. Use the parser.addini function inside pytest_addoption to add rp_uuid, rp_endpoint, rp_project and rp_launch ini option
b. Inside pytest_configure hook use config._inicache to add the parsed command line option to the ini value created above

import pytest
 
def pytest_addoption(parser):
    # Method to add the option to ini
    parser.addini("rp_uuid",'help',type="pathlist")
    parser.addini("rp_endpoint",'help',type="pathlist")
    parser.addini("rp_project",'help',type="pathlist")
    parser.addini("rp_launch",'help',type="pathlist")              
 
@pytest.hookimpl()
def pytest_configure(config):
    # Sets the launch name based on the marker selected.
    suite = config.getoption("markexpr")
    try:
        config._inicache["rp_uuid"]="f9ffd233-1c12-4bcc-87cd-89bc62865b4d"
        config._inicache["rp_endpoint"]="http://web.demo.reportportal.io"
        config._inicache["rp_project"]="default_personal"
        if suite == "gui_test":            
            config._inicache["rp_launch"]="gui_test"
        elif suite == "api_test":
            config._inicache["rp_launch"]="api_test" 
 
    except Exception,e:
        print (str(e))

5. Running the test
You can run the test in ReportalPortal using any of the markers as specified below

py.test -m gui_test --reportportal

Run_Test_ReportPortal

ReportPortal_Launch

Hurray, now we are able to run tests and analyze report for specific tests in ReportPortal. Happy testing…


References:

1. Working with plugins and conftest files
2. Adding INI file option
3. Allow to dynamically modify all ini-values from plugins


Avinash Shetty

I am a software tester with over 10 years of experience in software testing. Currently I am working at Qxf2 Services Bangalore. As a student of the context-driven approach to software testing I feel there is a lot to learn out there which keeps me very excited. My work has helped me gain good experience in different areas of testing like CRM, Web, Mobile and Database testing. I have good knowledge of building test scripts using Automation tools like Selenium and Appium using Java and Python. Beside testing I am a “Sports Fanatic” and love watching and playing sports.

Be First to Comment

Leave a Reply

Your email address will not be published.