Problem: If QA wants to make a difference with their testing and influence decision-making, we should convey our test results clearly. Hence test reporting becomes an important part of testing.
There are several test reporting tools available and recently many AI based test reporting tools are getting popular. Integrating with these reporting tools is useful. In this post, we will show you how to integrate a Python test which uses pytest as the test runner to integrate with a popular test reporting tool ReportPortal. ReportPortal helps to acquire, aggregate and analyze test reports to ascertain release health better. It also shows us trends over time.
Thank you Lea Anthony of Secure Code Warrior for introducing us to ReportPortal!
Why this post?
One of our clients was planning on using ReportPortal as the test reporting tool. The plain, vanilla integration of ReportPortal with pytest (which is the default test runner we use at Qxf2 Services) is straightforward and covered well by ReportPortal’s docs. The plugin for reporting test results of Pytest to the ReportPortal needs the pytest.ini to be updated at a root level. This was a problem for us on two counts:
a) we wanted to be able to have different ReportPortal projects (called “launch name” in ReportPortal) for different pytest markers. We know the marker only after the pytest.ini has been initialized. So the launch name used couldn’t be conditionally set in pytest.ini
b) we like the config parameters to be set somewhere other than declaring it directly in the pytest.ini file
How to integrate ReportPortal with pytest markers
To illustrate how to integrate ReportPortal with pytest markers, we will be running two tests which fall under API and GUI test groups. When we use the marker as gui_test, the GUI test should run and report to ReportPortal with a launch name as gui_test. When we use the marker as api_test, the API test should run and report to ReportPortal with a launch name as api_test.
Below are the steps needed to achieve this
1. Set up ReportPortal
2. Create two tests with markers
3. Set up pytest plugin for ReportPortal
4. Create a conftest
5. Run the test
1. Set up ReportPortal
You can go about downloading and setting up ReportPortal as mentioned or try their Demo server. I have used the demo server for this blog. Navigate to profile link and make a note of UUID, endpoint and project details.
2. Create two tests with markers
For the demo, we will create dummy tests with just the function names. We will add a couple of markers for gui and api tests.
Note: With pytest markers, you can restrict a test run to only run tests only with specific markers
import pytest @pytest.mark.gui_test() def test_gui(): print ("GUI Test") pass @pytest.mark.api_test() def test_api(): print ("API Test") pass |
3. Set up pytest plugin for ReportPortal
The agent-python-pytest plugin allows you to integrate pytest with ReportPortal. To install the plugin you can run
pip install pytest-reportportal |
The below example is the default way suggested for integrating with ReportPortal. You can skip this if you want to use a more flexible pytest hook implementation instead(discussed next).
Add the ReportPortal details like UUID, endpoint and project detail which you noted in step 2 and use it in pytest.ini file. Example:
[pytest] rp_uuid = fb586627-32be-47dd-93c1-678873458a5f rp_endpoint = http://192.168.1.10:8080 rp_project = user_personal rp_launch = AnyLaunchName |
Note: The UUID and other details mentioned above are for my Demo server and needs to be updated as per your ReportPortal details.
4. Creating a conftest
However, instead of the above example, we can to create a conftest file. If you use pytest regularly, you probably already have one! If not, create a conftest.py file where we can add the ini details conditionally which ReportPortal requires
a. Use the parser.addini function inside pytest_addoption to add rp_uuid, rp_endpoint, rp_project and rp_launch ini option
b. Inside pytest_configure hook use config._inicache to add the parsed command line option to the ini value created above
import pytest def pytest_addoption(parser): # Method to add the option to ini parser.addini("rp_uuid",'help',type="pathlist") parser.addini("rp_endpoint",'help',type="pathlist") parser.addini("rp_project",'help',type="pathlist") parser.addini("rp_launch",'help',type="pathlist") @pytest.hookimpl() def pytest_configure(config): # Sets the launch name based on the marker selected. suite = config.getoption("markexpr") try: config._inicache["rp_uuid"]="f9ffd233-1c12-4bcc-87cd-89bc62865b4d" config._inicache["rp_endpoint"]="http://web.demo.reportportal.io" config._inicache["rp_project"]="default_personal" if suite == "gui_test": config._inicache["rp_launch"]="gui_test" elif suite == "api_test": config._inicache["rp_launch"]="api_test" except Exception,e: print (str(e)) |
5. Running the test
You can run the test in ReportalPortal using any of the markers as specified below
py.test -m gui_test --reportportal |
Hurray, now we are able to run tests and analyze report for specific tests in ReportPortal. Happy testing…
References:
1. Working with plugins and conftest files
2. Adding INI file option
3. Allow to dynamically modify all ini-values from plugins
I am a dedicated quality assurance professional with a true passion for ensuring product quality and driving efficient testing processes. Throughout my career, I have gained extensive expertise in various testing domains, showcasing my versatility in testing diverse applications such as CRM, Web, Mobile, Database, and Machine Learning-based applications. What sets me apart is my ability to develop robust test scripts, ensure comprehensive test coverage, and efficiently report defects. With experience in managing teams and leading testing-related activities, I foster collaboration and drive efficiency within projects. Proficient in tools like Selenium, Appium, Mechanize, Requests, Postman, Runscope, Gatling, Locust, Jenkins, CircleCI, Docker, and Grafana, I stay up-to-date with the latest advancements in the field to deliver exceptional software products. Outside of work, I find joy and inspiration in sports, maintaining a balanced lifestyle.
Hello, how are you? I am in search of learning about python + pytests and your post was very helpful but, for now, I am having problems because there is an error in the realization of json, when making the call of py.test -m my_market –reportportal.
Do you have an email that we can talk to better and maybe you can support me with any suggested solutions? grateful … hugs
Hi Matheus,
Nice to know our posts helped you. Can you give more details on the error you are getting? My id is [email protected]
Hi, thanks … I sent you an email with error logs and scripts created for testing and configuration … Hugs
Hi ,
Currently we i try this , i get the result as below : Eventhough it is passing the results are not reflected in the report portal dashboard (i was using the demo version). Launch itself not reflected on the report portal.io. I have tried suppressing warnings as well – In that case the test cases pass but still the result is not sent to the report portal dashboard including the launch. Any help on this is well appreciated
py.test ./tests/markers.py –reportportal
==================================================== test session starts =====================================================
platform win32 — Python 3.6.5, pytest-6.1.1, py-1.9.0, pluggy-0.13.1
rootdir: D:\ReportPortal, configfile: pytest.ini
plugins: reportportal-5.0.5
collected 2 items
tests\markers.py .. [100%]
====================================================== warnings summary ======================================================
..\python\lib\site-packages\_pytest\config\__init__.py:1230
d:\python\lib\site-packages\_pytest\config\__init__.py:1230: PytestConfigWarning: Unknown config option: rp.endpoint
self._warn_or_fail_if_strict(“Unknown config option: {}\n”.format(key))
..\python\lib\site-packages\_pytest\config\__init__.py:1230
d:\python\lib\site-packages\_pytest\config\__init__.py:1230: PytestConfigWarning: Unknown config option: rp.launch
self._warn_or_fail_if_strict(“Unknown config option: {}\n”.format(key))
..\python\lib\site-packages\_pytest\config\__init__.py:1230
d:\python\lib\site-packages\_pytest\config\__init__.py:1230: PytestConfigWarning: Unknown config option: rp.project
self._warn_or_fail_if_strict(“Unknown config option: {}\n”.format(key))
..\python\lib\site-packages\_pytest\config\__init__.py:1230
d:\python\lib\site-packages\_pytest\config\__init__.py:1230: PytestConfigWarning: Unknown config option: rp.uuid
self._warn_or_fail_if_strict(“Unknown config option: {}\n”.format(key))
tests\markers.py:3
D:\ReportPortal\tests\markers.py:3: PytestUnknownMarkWarning: Unknown pytest.mark.gui_test – is this a typo? You can register custom marks to avoid this warning – for details, see https://docs.pytest.org/en/stable/mark.html
@pytest.mark.gui_test()
tests\markers.py:8
D:\ReportPortal\tests\markers.py:8: PytestUnknownMarkWarning: Unknown pytest.mark.api_test – is this a typo? You can register custom marks to avoid this warning – for details, see https://docs.pytest.org/en/stable/mark.html
@pytest.mark.api_test()
— Docs: https://docs.pytest.org/en/stable/warnings.html
=============================================== 2 passed, 6 warnings in 0.07s
Hi Thangam,
Based on the error message you have shared, I think you have not configured the Reportportal details such as rp_uuid, rp_endpoint correctly. I think you have given details such as rp.uuid, rp.endpoint in your `pytest.ini` file. You can refer to the blog as well below the documentation link for correcting your `pytest.ini` file.
https://pypi.org/project/pytest-reportportal/
Also, there are two PytestUnknowMarkUp warning, you may refer to the below link to fix those as well:
https://docs.pytest.org/en/stable/mark.html
I hope this helps you.
Regards,
Rahul
Hi Rahul, Thanks! The warnings went off when i downgraded pytest to version 5 . The tests passes but still the launch is not appearing in the report portal . I am using the demo version
The issue got resolved, it was due to the token change in report portal for the demo version
Hi Thangam,
Good to know your issue got resolved.
Regards,
Rahul
Thanks you very much. Your post helped me get my ReportPortal instance gather lots of different dynamic values I wanted to pass in. Thank you.
Hi ,
Thanks for the wonderful blog. I have implemented the aforementioned RP changes for pytest in my project but I am not able to figure out the order of giving my dynamic params for the same.
for e.g when I use the command line arguments which are needed to my project
pytest -hostport https://xxxx.com/sabrix/services/rest -userName admin -pwd admin -environment qa -m markername –reportportal
ERROR: -o/–override-ini expects option=value style (got: ‘stport’).
When I just use the base pytest command
pytest -m markername –reportportal
getting the stderr to use the required params
kindly help me to give the order of parameters for the above, appreciate your help in this regarding
Thanks,
Dinesh
Hi Dinesh,
I think it will be easier for you if you implement, params as pytest fixtures in the conftest.py file. The report portal implementation example is already available in Qxf2’s Page Object Model framework. You can have a look at the framework at below:
https://github.com/qxf2/qxf2-page-object-model
I hope this will help you to implement reportportal implementation.
Regards,
Rahul
Hi, i tried your code in my environtment. It gave me a error everytime i run it. This is the error:
mock\mock_test.py:3
C:\Users\karnor\Documents\agent-python-pytest-master\mock\mock_test.py:3: PytestUnknownMarkWarning: Unknown pytest.mark.gui_test – is this a typo? You can register custom marks to avoid this warning – for details, see https://docs.pytest.org/en/stable/mark.html
pytest.mark.gui_test()
mock\mock_test.py:8
C:\Users\karnor\Documents\agent-python-pytest-master\mock\mock_test.py:8: PytestUnknownMarkWarning: Unknown pytest.mark.api_test – is this a typo? You can register custom marks to avoid this warning – for details, see https://docs.pytest.org/en/stable/mark.html
pytest.mark.api_test()
— Docs: https://docs.pytest.org/en/stable/warnings.html
It seems that the gui_test file is the problem but i dont know what causes it, maybe you could help. Thanks in advance!!
HI,
To properly handle this you need to register the custom marker. In the above provided warning as well says the same. In pytest.ini file place the following inside of it.
markers = gui_test: mark a test as a gui_test.
Regards,
Raghava
HI,
To properly handle this you need to register the custom marker. In the above provided warning as well says the same. In pytest.ini file place the following inside of it.
markers = gui_test: mark a test as a gui_test.
Regards,
Raghava
Hello Avinash, Thanks for nice blogs. I always enjoy reading them. I am just curious to know so far how is your experience using Reporting Portal? Does it bring any additional tangible benefits compared to traditional reporting tools such as Allure?
I have integrated pytest with report portal. Launches are showing up in report portal as well but the I’m facing a issue here, total tests I have executed is 12 but in the report portal it is only showing as 4, any help?
Can you please share your masked test output from the console?
Could it be that 8/12 tests were deselected because you used a marker that excluded those tests?