We wanted a good way to report the test results from our automation framework. So, when Tesults approached us to integrate with our Selenium, Python-based automation framework we were pleased. In this post, we are going to show you how automation can update test results directly on to Tesults.
Why this post?
If you have ever built a automated testing framework from scratch, we are sure you would have realized that reporting results is an extremely challenging task. We have struggled with generating meaningful reports that summarize the data into useful information. Continuous integration tools like Jenkins and CircleCI are great for engineers – but non-technical stakeholders prefer something more visual. So far, Qxf2 has been using TestRail to record and share the results of our automation runs. While our approach has worked to some degree, it has the overhead of documenting a test case on TestRail for each automated test case that we write. We are trying to figure out other solutions that can help us. In the coming weeks, we will share several approaches we have been trying. In this post, we will write about a solution/product that is focused exclusively on solving the problems around the storing and reporting of automated test results.
Some background
Ajeet Dhaliwal‘s team from Tesults integrated Tesults into our framework. Thanks, Ajeet for initiating and completing the integration in a short duration. I will be sharing the changes Ajeet came up for the integration. This would help you in case you are planning to integrate Tesults into your framework. If you are looking to avoid all the hard work of writing an automation framework from scratch, you can try Qxf2’s open-sourced automation framework.
While we are yet to use Tesults at a client (we learnt of it just a couple of months ago!), we are sure our integration with Tesults will help us report our automated checks to clients and stakeholders in a better way. It will also make it much easier for us to maintain our tests without having the overhead to go and update TestRail every time we write a new test.
About Tesults
Tesults is a web-based automated test result and build status reporting application for teams. You can store, report and analyze results data including logs, screen captures, historical trends, regression spotting and team triage of failing tests and build failures.
Report automation results to Tesults using Python
We’ll need to perform the following steps to report test results automatically to Tesults.
1. Sign up with Tesults
2. Create a project
3. Install Tesults and import it into your code
4. Modify your test script to report to Tesults
1. Sign up
You can sign up for Tesults using this link. They offer a free account if you want to try it.
2. Create a project
Once you sign up you need to create a New project. You will be provided with a token that is used to upload test results to the default target. We saved this token in a conf file which is required when you post data to Tesults. You can get more details on this step here
3. Install Tesults and import it into your code
The Qxf2 Selenium automation framework uses Python. We will use pip to install Tesults
pip install tesults |
Once you have installed Tesults you need to import it to your project
import tesults |
For more details on installation and configuration refer to this link
4. Modify your test script to report to Tesults
The Tesults.py utility file can help you to post data to Tesults.
import tesults import conf.tesults_conf as conf_file #The tesults_conf file has the project token. cases = [] def add_test_case(data): cases.append(data) def post_results_to_tesults (): token = conf_file.target_token_default # uses default token unless otherwise specified data = { 'target': token, 'results': { 'cases': cases } } print ('-----Tesults output-----') if len(data['results']['cases']) > 0: print (data) print('Uploading results to Tesults...') ret = tesults.results(data) print ('success: ' + str(ret['success'])) print ('message: ' + str(ret['message'])) print ('warnings: ' + str(ret['warnings'])) print ('errors: ' + str(ret['errors'])) else: print ('No test results.') |
It has two methods as described below.
a. add_test_case(): All the data which needs to be reported is added and appended to a list using this method.
We will be calling this method using the add_tesults_case() method defined in the Base_Page.py. The data param which has to be sent is a dictionary containing your build and test results.
def add_tesults_case(self, name, desc, suite, result_flag, msg='', files=[], params={}, custom={}): if self.tesults_flag is True: result = "unknown" failReason = "" if result_flag == True: result = "pass" if result_flag == False: result = "fail" failReason = msg for image in self.images: files.append(self.screenshot_dir + os.sep + image + '.png') self.images = [] caseObj = {'name': name, 'suite': suite, 'desc': desc, 'result': result, 'reason': failReason, 'files': files, 'params': params} for key, value in custom.iteritems(): caseObj[key] = str(value) Tesults.add_test_case(caseObj) |
Here is a list of test case properties used for reporting results to Tesults. name (Name of the test), result (Result of the test: pass, fail, unknown), suite (Suite the test belongs to), desc (Description of the test), reason (Reason for the test failure), params (Parameters of the test), files (Files such as logs, screenshots etc)
Now we can call the add_tesults_case from our test directly whenever we want to add the result which has to be reported to Tesults later.
test_obj.add_tesults_case("Set Name", "Sets the name in the form", "test_example_form", result_flag, "Failed to set name: %s \nOn url: %s\n"%(name,test_obj.get_current_url())) |
Note: We are only passing some test case properties as required. name and result are required fields, rest are optional.
b. post_results_to_tesults(): Tesults provides a “results” method to upload your test results to Tesults.
tesults.results(data) |
The post_results_to_tesults() will be calling this method to post the report to Tesults. This call returns a dictionary indicating success or failure. We have used it to print the results as given in first code block.
It’s totally up to you where you can call this method. But its usually done in the teardown functions of a test framework or as a post-build step. Since our framework uses pytest we are calling this method as part of pytest_terminal_summary() method.
if pytest.config.getoption("--tesults").lower() == 'y': Tesults.post_results_to_tesults() |
Here is a sample view of how the report looks in Tesults for one of our test runs.
Hope this post helped you to get an idea how to integrate Tesults with an automation Framework. You can get all this code in our open-sourced Python test automation framework based on the page object pattern. We strongly recommend you check it out!
I am a dedicated quality assurance professional with a true passion for ensuring product quality and driving efficient testing processes. Throughout my career, I have gained extensive expertise in various testing domains, showcasing my versatility in testing diverse applications such as CRM, Web, Mobile, Database, and Machine Learning-based applications. What sets me apart is my ability to develop robust test scripts, ensure comprehensive test coverage, and efficiently report defects. With experience in managing teams and leading testing-related activities, I foster collaboration and drive efficiency within projects. Proficient in tools like Selenium, Appium, Mechanize, Requests, Postman, Runscope, Gatling, Locust, Jenkins, CircleCI, Docker, and Grafana, I stay up-to-date with the latest advancements in the field to deliver exceptional software products. Outside of work, I find joy and inspiration in sports, maintaining a balanced lifestyle.
