ReportPortal integration with pytest and pytest markers

Problem: If QA wants to make a difference with their testing and influence decision-making, we should convey our test results clearly. Hence test reporting becomes an important part of testing.
 
There are several test reporting tools available and recently many AI based test reporting tools are getting popular. Integrating with these reporting tools is useful. In this post, we will show you how to integrate a Python test which uses pytest as the test runner to integrate with a popular test reporting tool ReportPortal. ReportPortal helps to acquire, aggregate and analyze test reports to ascertain release health better. It also shows us trends over time.
 
Thank you Lea Anthony of Secure Code Warrior for introducing us to ReportPortal!


Why this post?

One of our clients was planning on using ReportPortal as the test reporting tool. The plain, vanilla integration of ReportPortal with pytest (which is the default test runner we use at Qxf2 Services) is straightforward and covered well by ReportPortal’s docs. The plugin for reporting test results of Pytest to the ReportPortal needs the pytest.ini to be updated at a root level. This was a problem for us on two counts:
a) we wanted to be able to have different ReportPortal projects (called “launch name” in ReportPortal) for different pytest markers. We know the marker only after the pytest.ini has been initialized. So the launch name used couldn’t be conditionally set in pytest.ini
b) we like the config parameters to be set somewhere other than declaring it directly in the pytest.ini file


How to integrate ReportPortal with pytest markers

To illustrate how to integrate ReportPortal with pytest markers, we will be running two tests which fall under API and GUI test groups. When we use the marker as gui_test, the GUI test should run and report to ReportPortal with a launch name as gui_test. When we use the marker as api_test, the API test should run and report to ReportPortal with a launch name as api_test.

Below are the steps needed to achieve this
1. Set up ReportPortal
2. Create two tests with markers
3. Set up pytest plugin for ReportPortal
4. Create a conftest
5. Run the test

1. Set up ReportPortal
You can go about downloading and setting up ReportPortal as mentioned or try their Demo server. I have used the demo server for this blog. Navigate to profile link and make a note of UUID, endpoint and project details.

ReportPortal_Profile
2. Create two tests with markers

For the demo, we will create dummy tests with just the function names. We will add a couple of markers for gui and api tests.
Note: With pytest markers, you can restrict a test run to only run tests only with specific markers

import pytest
 
@pytest.mark.gui_test()
def test_gui():
    print ("GUI Test")
    pass
 
@pytest.mark.api_test()
def test_api():
    print ("API Test")
    pass

3. Set up pytest plugin for ReportPortal
The agent-python-pytest plugin allows you to integrate pytest with ReportPortal. To install the plugin you can run

pip install pytest-reportportal

The below example is the default way suggested for integrating with ReportPortal. You can skip this if you want to use a more flexible pytest hook implementation instead(discussed next).

Add the ReportPortal details like UUID, endpoint and project detail which you noted in step 2 and use it in pytest.ini file. Example:

[pytest]
rp_uuid = fb586627-32be-47dd-93c1-678873458a5f
rp_endpoint = http://192.168.1.10:8080
rp_project = user_personal
rp_launch = AnyLaunchName

Note: The UUID and other details mentioned above are for my Demo server and needs to be updated as per your ReportPortal details.

4. Creating a conftest
However, instead of the above example, we can to create a conftest file. If you use pytest regularly, you probably already have one! If not, create a conftest.py file where we can add the ini details conditionally which ReportPortal requires

a. Use the parser.addini function inside pytest_addoption to add rp_uuid, rp_endpoint, rp_project and rp_launch ini option
b. Inside pytest_configure hook use config._inicache to add the parsed command line option to the ini value created above

import pytest
 
def pytest_addoption(parser):
    # Method to add the option to ini
    parser.addini("rp_uuid",'help',type="pathlist")
    parser.addini("rp_endpoint",'help',type="pathlist")
    parser.addini("rp_project",'help',type="pathlist")
    parser.addini("rp_launch",'help',type="pathlist")              
 
@pytest.hookimpl()
def pytest_configure(config):
    # Sets the launch name based on the marker selected.
    suite = config.getoption("markexpr")
    try:
        config._inicache["rp_uuid"]="f9ffd233-1c12-4bcc-87cd-89bc62865b4d"
        config._inicache["rp_endpoint"]="http://web.demo.reportportal.io"
        config._inicache["rp_project"]="default_personal"
        if suite == "gui_test":            
            config._inicache["rp_launch"]="gui_test"
        elif suite == "api_test":
            config._inicache["rp_launch"]="api_test" 
 
    except Exception,e:
        print (str(e))

5. Running the test
You can run the test in ReportalPortal using any of the markers as specified below

py.test -m gui_test --reportportal

Run_Test_ReportPortal

ReportPortal_Launch

Hurray, now we are able to run tests and analyze report for specific tests in ReportPortal. Happy testing…


References:

1. Working with plugins and conftest files
2. Adding INI file option
3. Allow to dynamically modify all ini-values from plugins


14 thoughts on “ReportPortal integration with pytest and pytest markers

  1. Hello, how are you? I am in search of learning about python + pytests and your post was very helpful but, for now, I am having problems because there is an error in the realization of json, when making the call of py.test -m my_market –reportportal.
    Do you have an email that we can talk to better and maybe you can support me with any suggested solutions? grateful … hugs

  2. Hi, thanks … I sent you an email with error logs and scripts created for testing and configuration … Hugs

  3. Hi ,

    Currently we i try this , i get the result as below : Eventhough it is passing the results are not reflected in the report portal dashboard (i was using the demo version). Launch itself not reflected on the report portal.io. I have tried suppressing warnings as well – In that case the test cases pass but still the result is not sent to the report portal dashboard including the launch. Any help on this is well appreciated
    py.test ./tests/markers.py –reportportal
    ==================================================== test session starts =====================================================
    platform win32 — Python 3.6.5, pytest-6.1.1, py-1.9.0, pluggy-0.13.1
    rootdir: D:\ReportPortal, configfile: pytest.ini
    plugins: reportportal-5.0.5
    collected 2 items

    tests\markers.py .. [100%]

    ====================================================== warnings summary ======================================================
    ..\python\lib\site-packages\_pytest\config\__init__.py:1230
    d:\python\lib\site-packages\_pytest\config\__init__.py:1230: PytestConfigWarning: Unknown config option: rp.endpoint

    self._warn_or_fail_if_strict(“Unknown config option: {}\n”.format(key))

    ..\python\lib\site-packages\_pytest\config\__init__.py:1230
    d:\python\lib\site-packages\_pytest\config\__init__.py:1230: PytestConfigWarning: Unknown config option: rp.launch

    self._warn_or_fail_if_strict(“Unknown config option: {}\n”.format(key))

    ..\python\lib\site-packages\_pytest\config\__init__.py:1230
    d:\python\lib\site-packages\_pytest\config\__init__.py:1230: PytestConfigWarning: Unknown config option: rp.project

    self._warn_or_fail_if_strict(“Unknown config option: {}\n”.format(key))

    ..\python\lib\site-packages\_pytest\config\__init__.py:1230
    d:\python\lib\site-packages\_pytest\config\__init__.py:1230: PytestConfigWarning: Unknown config option: rp.uuid

    self._warn_or_fail_if_strict(“Unknown config option: {}\n”.format(key))

    tests\markers.py:3
    D:\ReportPortal\tests\markers.py:3: PytestUnknownMarkWarning: Unknown pytest.mark.gui_test – is this a typo? You can register custom marks to avoid this warning – for details, see https://docs.pytest.org/en/stable/mark.html
    @pytest.mark.gui_test()

    tests\markers.py:8
    D:\ReportPortal\tests\markers.py:8: PytestUnknownMarkWarning: Unknown pytest.mark.api_test – is this a typo? You can register custom marks to avoid this warning – for details, see https://docs.pytest.org/en/stable/mark.html
    @pytest.mark.api_test()

    — Docs: https://docs.pytest.org/en/stable/warnings.html
    =============================================== 2 passed, 6 warnings in 0.07s

    1. Hi Thangam,

      Based on the error message you have shared, I think you have not configured the Reportportal details such as rp_uuid, rp_endpoint correctly. I think you have given details such as rp.uuid, rp.endpoint in your `pytest.ini` file. You can refer to the blog as well below the documentation link for correcting your `pytest.ini` file.
      https://pypi.org/project/pytest-reportportal/

      Also, there are two PytestUnknowMarkUp warning, you may refer to the below link to fix those as well:
      https://docs.pytest.org/en/stable/mark.html

      I hope this helps you.

      Regards,
      Rahul

      1. Hi Rahul, Thanks! The warnings went off when i downgraded pytest to version 5 . The tests passes but still the launch is not appearing in the report portal . I am using the demo version

      2. The issue got resolved, it was due to the token change in report portal for the demo version

  4. Thanks you very much. Your post helped me get my ReportPortal instance gather lots of different dynamic values I wanted to pass in. Thank you.

  5. Hi ,
    Thanks for the wonderful blog. I have implemented the aforementioned RP changes for pytest in my project but I am not able to figure out the order of giving my dynamic params for the same.

    for e.g when I use the command line arguments which are needed to my project
    pytest -hostport https://xxxx.com/sabrix/services/rest -userName admin -pwd admin -environment qa -m markername –reportportal

    ERROR: -o/–override-ini expects option=value style (got: ‘stport’).

    When I just use the base pytest command
    pytest -m markername –reportportal
    getting the stderr to use the required params
    kindly help me to give the order of parameters for the above, appreciate your help in this regarding

    Thanks,
    Dinesh

    1. Hi Dinesh,

      I think it will be easier for you if you implement, params as pytest fixtures in the conftest.py file. The report portal implementation example is already available in Qxf2’s Page Object Model framework. You can have a look at the framework at below:

      https://github.com/qxf2/qxf2-page-object-model

      I hope this will help you to implement reportportal implementation.

      Regards,
      Rahul

  6. Hi, i tried your code in my environtment. It gave me a error everytime i run it. This is the error:
    mock\mock_test.py:3
    C:\Users\karnor\Documents\agent-python-pytest-master\mock\mock_test.py:3: PytestUnknownMarkWarning: Unknown pytest.mark.gui_test – is this a typo? You can register custom marks to avoid this warning – for details, see https://docs.pytest.org/en/stable/mark.html
    pytest.mark.gui_test()

    mock\mock_test.py:8
    C:\Users\karnor\Documents\agent-python-pytest-master\mock\mock_test.py:8: PytestUnknownMarkWarning: Unknown pytest.mark.api_test – is this a typo? You can register custom marks to avoid this warning – for details, see https://docs.pytest.org/en/stable/mark.html
    pytest.mark.api_test()

    — Docs: https://docs.pytest.org/en/stable/warnings.html
    It seems that the gui_test file is the problem but i dont know what causes it, maybe you could help. Thanks in advance!!

    1. HI,

      To properly handle this you need to register the custom marker. In the above provided warning as well says the same. In pytest.ini file place the following inside of it.
      markers = gui_test: mark a test as a gui_test.

      Regards,
      Raghava

    2. HI,
      To properly handle this you need to register the custom marker. In the above provided warning as well says the same. In pytest.ini file place the following inside of it.
      markers = gui_test: mark a test as a gui_test.

      Regards,
      Raghava

Leave a Reply

Your email address will not be published.