Tesults: better reporting for automated tests

We wanted a good way to report the test results from our automation framework. So, when Tesults approached us to integrate with our Selenium, Python-based automation framework we were pleased. In this post, we are going to show you how automation can update test results directly on to Tesults.


Why this post?

If you have ever built a automated testing framework from scratch, we are sure you would have realized that reporting results is an extremely challenging task. We have struggled with generating meaningful reports that summarize the data into useful information. Continuous integration tools like Jenkins and CircleCI are great for engineers – but non-technical stakeholders prefer something more visual. So far, Qxf2 has been using TestRail to record and share the results of our automation runs. While our approach has worked to some degree, it has the overhead of documenting a test case on TestRail for each automated test case that we write. We are trying to figure out other solutions that can help us. In the coming weeks, we will share several approaches we have been trying. In this post, we will write about a solution/product that is focused exclusively on solving the problems around the storing and reporting of automated test results.


Some background

Ajeet Dhaliwal‘s team from Tesults integrated Tesults into our framework. Thanks, Ajeet for initiating and completing the integration in a short duration. I will be sharing the changes Ajeet came up for the integration. This would help you in case you are planning to integrate Tesults into your framework. If you are looking to avoid all the hard work of writing an automation framework from scratch, you can try Qxf2’s open-sourced automation framework.

While we are yet to use Tesults at a client (we learnt of it just a couple of months ago!), we are sure our integration with Tesults will help us report our automated checks to clients and stakeholders in a better way. It will also make it much easier for us to maintain our tests without having the overhead to go and update TestRail every time we write a new test.


About Tesults

Tesults is a web-based automated test result and build status reporting application for teams. You can store, report and analyze results data including logs, screen captures, historical trends, regression spotting and team triage of failing tests and build failures.


Report automation results to Tesults using Python

We’ll need to perform the following steps to report test results automatically to Tesults.
1. Sign up with Tesults
2. Create a project
3. Install Tesults and import it into your code
4. Modify your test script to report to Tesults

 
1. Sign up
You can sign up for Tesults using this link. They offer a free account if you want to try it.

2. Create a project
Once you sign up you need to create a New project. You will be provided with a token that is used to upload test results to the default target. We saved this token in a conf file which is required when you post data to Tesults. You can get more details on this step here

3. Install Tesults and import it into your code
The Qxf2 Selenium automation framework uses Python. We will use pip to install Tesults

pip install tesults

Once you have installed Tesults you need to import it to your project

import tesults

For more details on installation and configuration refer to this link

4. Modify your test script to report to Tesults
The Tesults.py utility file can help you to post data to Tesults.

import tesults
import conf.tesults_conf as conf_file #The tesults_conf file has the project token.
 
cases = []
 
def add_test_case(data):
    cases.append(data)
 
def post_results_to_tesults ():
    token = conf_file.target_token_default # uses default token unless otherwise specified
    data = {
        'target': token,
        'results': { 'cases': cases }
    }
    print ('-----Tesults output-----')
    if len(data['results']['cases']) > 0:
        print (data)
        print('Uploading results to Tesults...')
        ret = tesults.results(data)
        print ('success: ' + str(ret['success']))
        print ('message: ' + str(ret['message']))
        print ('warnings: ' + str(ret['warnings']))
        print ('errors: ' + str(ret['errors']))
    else:
        print ('No test results.')

It has two methods as described below.
a. add_test_case(): All the data which needs to be reported is added and appended to a list using this method.

We will be calling this method using the add_tesults_case() method defined in the Base_Page.py. The data param which has to be sent is a dictionary containing your build and test results.

def add_tesults_case(self, name, desc, suite, result_flag, msg='', files=[], params={}, custom={}):
        if self.tesults_flag is True:
            result = "unknown"
            failReason = ""
            if result_flag == True:
                result = "pass"
            if result_flag == False:
                result = "fail"
                failReason = msg
            for image in self.images:
                files.append(self.screenshot_dir + os.sep + image + '.png')
            self.images = []
            caseObj = {'name': name, 'suite': suite, 'desc': desc, 'result': result, 'reason': failReason, 'files': files, 'params': params}
            for key, value in custom.iteritems():
                caseObj[key] = str(value)
            Tesults.add_test_case(caseObj)

Here is a list of test case properties used for reporting results to Tesults. name (Name of the test), result (Result of the test: pass, fail, unknown), suite (Suite the test belongs to), desc (Description of the test), reason (Reason for the test failure), params (Parameters of the test), files (Files such as logs, screenshots etc)

Now we can call the add_tesults_case from our test directly whenever we want to add the result which has to be reported to Tesults later.

test_obj.add_tesults_case("Set Name", "Sets the name in the form", "test_example_form", result_flag, "Failed to set name: %s \nOn url: %s\n"%(name,test_obj.get_current_url()))

Note: We are only passing some test case properties as required. name and result are required fields, rest are optional.

b. post_results_to_tesults(): Tesults provides a “results” method to upload your test results to Tesults.

tesults.results(data)

The post_results_to_tesults() will be calling this method to post the report to Tesults. This call returns a dictionary indicating success or failure. We have used it to print the results as given in first code block.

It’s totally up to you where you can call this method. But its usually done in the teardown functions of a test framework or as a post-build step. Since our framework uses pytest we are calling this method as part of pytest_terminal_summary() method.

if pytest.config.getoption("--tesults").lower() == 'y':
        Tesults.post_results_to_tesults()

Here is a sample view of how the report looks in Tesults for one of our test runs.

Tesults_Report


Hope this post helped you to get an idea how to integrate Tesults with an automation Framework. You can get all this code in our open-sourced Python test automation framework based on the page object pattern. We strongly recommend you check it out!
 


Avinash Shetty

I am a software tester with over 10 years of experience in software testing. Currently I am working at Qxf2 Services Bangalore. As a student of the context-driven approach to software testing I feel there is a lot to learn out there which keeps me very excited. My work has helped me gain good experience in different areas of testing like CRM, Web, Mobile and Database testing. I have good knowledge of building test scripts using Automation tools like Selenium and Appium using Java and Python. Beside testing I am a “Sports Fanatic” and love watching and playing sports.

Be First to Comment

Leave a Reply

Your email address will not be published.