Reporting automation results to TestRail using Python

We have been hard at work integrating our Python based GUI automation framework (that uses the Page Object pattern) with a test case management tool for reporting purposes. In this post we will show you how to report the results of your automated checks to a visual test case management tool – TestRail. TestRail offers a free trial – so you can sign up for free and follow this post.


Why this post?

In most automated checks, we are able to judge the outcome as ‘pass’ or ‘fail’ by comparing the expected result to the actual result. Tracking, reporting and sharing the test results is important. Excel and email rarely scale and need a lot of maintenance. Continuous Integration servers are great for engineers – but non-technical stakeholders prefer something more visual. At Qxf2, we love TestRail – a visual test case management software. We got you started with TestRail in an earlier post. In this post, we are going to show you how automation can update test results directly on to TestRail.


Report automation results to TestRail using Python

We’ll need to perform the following steps to report test results automatically to TestRail
1. Get the TestRail API
2. Connect (authentication) to your TestRail instance
3. Write the logic to update test results to TestRail
4. Identify your test case id on TestRail
5. Modify your test script to report to TestRail
 
 
1. Get the TestRail API
TestRail provides a useful REST API for automated tests to directly report results on to TestRail. The API provides the functionality to authenticate API requests, provides seamless JSON encoding/decoding and has generic support for read and write requests. We will write some useful wrappers around TestRail’s official API binding for Python.
Before we proceed, please download the testrail.py from here and follow the installation instructions.

2. Connect to your TestRail instance
We prefer separating any credentials we use into separate files. For this tutorial, we have created testrail.env that looks something like this:
testrail.env

TESTRAIL_URL=https://.testrail.com
TESTRAIL_USER=test@abc.xyz
TESTRAIL_PASSWORD=test

We then created a file that wraps around TestRail’s API called TestRail.py. In this file we implemented a method to connect to the TestRail instance.

"""
TestRail integration
"""
import dotenv,os,testrail,Conf_Reader
 
def get_testrail_client():
    "Get the TestRail account credentials from the testrail.env file"
    testrail_file = os.path.join(os.path.dirname(__file__),'testrail.env')
    #Get the TestRail Url
    testrail_url = Conf_Reader.get_value(testrail_file,'TESTRAIL_URL')
    client = testrail.APIClient(testrail_url)
    #Get and set the TestRail User and Password
    client.user = Conf_Reader.get_value(testrail_file,'TESTRAIL_USER')
    client.password = Conf_Reader.get_value(testrail_file,'TESTRAIL_PASSWORD')
 
    return client

Now we have an object that can connect to your TestRail instance!

3. Write the logic to update test results to TestRail
TestRail needs to know two things to go an update a test case result:
a. the test run id
b. the test case id

We need both because we can have multiple test runs for the same set of test cases. So our script should be able to update the test results into TestRail for a particular test run and specific test case. For this we need to know the test case id and the test run id. We implemented the update_testrail method. This method can be called from the test script with appropriate parameters so that the test results are recorded in TestRail for the specific case id and run id.

def update_testrail(case_id,run_id,result_flag,msg=""):
    "Update TestRail for a given run_id and case_id"
    update_flag = False
    #Get the TestRail client account details
    client = get_testrail_client()
 
    #Update the result in TestRail using send_post function. 
    #Parameters for add_result_for_case is the combination of runid and case id. 
    #status_id is 1 for Passed, 2 For Blocked, 4 for Retest and 5 for Failed
    status_id = 1 if result_flag is True else 5
 
    if run_id is not None:
        try:
            result = client.send_post(
                'add_result_for_case/%s/%s'%(run_id,case_id),
                {'status_id': status_id, 'comment': msg })
        except Exception,e:
            print 'Exception in update_testrail() updating TestRail.'
            print 'PYTHON SAYS: '
            print e
        else:
            print 'Updated test result for case: %s in test run: %s with msg:%s'%(case_id,run_id,msg)
 
    return update_flag

NOTE: Test case ids stay constant across test runs in TestRail. However each test run is assigned a unique id. To automatically obtain your run id based on your test run name, you can implement something similar to this code snippet:

    def get_project_id(project_name):
        "Get the project ID using project name"
        client = get_testrail_client()
        project_id=None
        projects = client.send_get('get_projects')
        for project in projects:
            if project['name'] == project_name: 
                project_id = project['id']
                #project_found_flag=True
                break
        return project_id
 
    def get_run_id(test_run_name,project_name):
        "Get the run ID using test name and project name"
        run_id=None
        client = get_testrail_client()
        project_id = get_project_id(project_name)
        try:
            test_runs = client.send_get('get_runs/%s'%(project_id))
        except Exception,e:
            print 'Exception in update_testrail() updating TestRail.'
            print 'PYTHON SAYS: '
            print e
            return None
        else:
            for test_run in test_runs:
                 if test_run['name'] == test_run_name:
                     run_id = test_run['id']
                     break
            return run_id

4. Identify your test case id on TestRail
To obtain the test case id, simply visit your project on TestRail, navigate to Test Cases section and look for the integer in the ID column next to your test case.

TestRail case ids

5. Modify your test script to report to TestRail
Now you can modify your existing test scripts to report to TestRail. Here is a code sample that shows you how to use the method you wrote earlier:
Test Script

#Code to automate test case goes here
.
.
#Update TestRail
    case_id = 125 #Case id of your test case 
    if (result_flag):
        msg = "Successfully updated the example form"
    else:
        msg = "Failed to update the example form"
    Test_Rail.update_testrail(case_id,test_run_id,result_flag,msg=msg)

Hope this post helps you offload the manual task of updating the test results in TestRail! Get all this code for free in our open-sourced Python test automation framework based on the page object pattern. We strongly recommend you check it out!


Subscribe to our weekly Newsletter


View a sample



Vrushali Toshniwal

My journey as a tester started at Sun Microsystems (now Oracle). I was part of the testing and sustaining team for the Portal Server and Identity Management products. My first assignment was to test the Rewriter module. I enjoyed understanding the big picture, writing test cases, finding bugs and sometimes suggesting the fix too! I was hooked onto testing. Testing felt natural and intuitive to me. I am technically inclined and can write automation in Java, C++, Perl and Python. I am well versed with SilkTest, Selenium, Appium and Selendroid. I am a Computer Science graduate from BITS-Pilani. I love travelling and listening to music.

40 Comments

  1. Vaishali Hegde said:

    Hi,

    I wanted to upload test cases to Test Rail in the test rail xml format. Is there any python modules to conevrt excel to Test Raril xml format?

    January 7, 2016
    Reply
  2. Shakti said:

    Hi ,
    Could you please help me reporting automation results to TestRail using java.

    January 19, 2016
    Reply
  3. noor said:

    PLease help me in setup testrail tool integration with selenium tool for UI automation.
    New in this field

    June 14, 2016
    Reply
    • Shivahari P Shivahari P said:

      Hi Noor,
      Can you be more specific about the issue that is blocking you?. Will allow us to help you better.

      Thanks

      June 14, 2016
      Reply
  4. Dinesh Kumar said:

    I’ve got Selenium automation tests using C# with xUnit test framework. When the tests are run the results are generated into a xUnit standard XML file. I am looking at the ways to somehow import the results from XML file into the TestRail so that I have a central repo for all test runs and I can generate meaningful reports inside TestRail.

    I know there’s an API I can use but creating methods inside the tests to post each test result is looking very cumbersome. Instead it would be really great if I can import the XML file into TestRail.

    Please help.

    June 21, 2016
    Reply
    • Dinesh, it doesn’t seem like there is anything like this already in existence. Gurock software is very responsive to its users. So you could try asking on their forums and sometimes they may give you a beta script to try out.

      The other alternative is to write your own custom mapper from XML to TestRail and use it. But I do not recommend this approach because it is going to be heavy on maintenance.

      June 24, 2016
      Reply
  5. G said:

    Hi,

    anyone can give me an example for the login page, how to read the data from the configuration file and call it into the login test case?
    I have created three files config.pf, login_helper.py, and login_testcase but I am really confusing how to call or integrate between these files?
    please help me on it…

    June 23, 2016
    Reply
      • G said:

        Thanks for the information….

        And I want to know that same concept in selenium web driver using python?
        why “import dotenv” is using ?

        June 23, 2016
        Reply
        • Shivahari P Shivahari P said:

          dotenv is a module in python to read key,value from a file and add them to the environment variables. i.e,dotenv.load_dotenv(conf) reads the key,value pairs from the conf file and adds them to the environment variable so that they can be fetched using value = os.environ[key].

          June 23, 2016
          Reply
  6. G said:

    ok

    Thank u so much

    June 23, 2016
    Reply
  7. Mina said:

    I have tried to use “close_run”, and according to the API(v2) doc, there is only two arguments for the method, while I sent this request, only got the info below:
    TypeError: send_post() takes exactly 3 arguments (2 given)

    what does the third argument look like? Let me show the api doc:
    POST index.php?/api/v2/close_run/:run_id

    September 2, 2016
    Reply
  8. Liza said:

    What is the Conf_Reader module in the step 2?

    September 5, 2016
    Reply
    • Shivahari P Shivahari P said:

      Hi Liza
      To avoid hard coding values into test scripts, we define them in conf files and them read them into the test. Conf_Reader is a custom module we wrote to fetch values from the conf files. So in this example, we wrote the TestRail URL, TestRail username, TestRail password into the env file and read it from it. To follow along without a Conf_Reader, just declare/hardcode the values of the TestRail URL, TestRail username, TestRail password in the get_testrail_client() method.

      September 7, 2016
      Reply
  9. Yago said:

    Hi,

    How can I update this result_flag? I have a few ideas but I would like to know your method for this.

    Best regards, Yago.

    October 27, 2016
    Reply
    • Shivahari P Shivahari P said:

      Hi Yago,
      I would enclose statements in a try-catch block such that the result_flag returns True if the WebDriver is able to perform an action.
      I would then validate the result_flag using conditional statements in test and update TestRail accordingly.
      Thanks

      November 2, 2016
      Reply
  10. VRR said:

    Hi ,

    I want to add project, Create test suite ,update testsuite .Is there any code available using python scripting.Please post me the code.

    December 21, 2016
    Reply
    • VRR, you can use this link as your reference: API reference: http://docs.gurock.com/testrail-api2/start

      For example, to add a project, your code would look something like this:

       def create_new_project(self,new_project_name,project_description,show_announcement,suite_mode):
              "Create a new project if it does not already exist"
              project_id = self.get_project_id(new_project_name)
              if project_id is None:
                  try:
                      result = self.client.send_post('add_project',
                                                     {'name': new_project_name,
                                                      'announcement': project_description,
                                                      'show_announcement': show_announcement,
                                                      'suite_mode': suite_mode,})
                  except Exception,e:
                      print 'Exception in create_new_project() creating new project.'
                      print 'PYTHON SAYS: '
                      print e
              else:
                  print "Project already exists %s"%new_project_name
      December 25, 2016
      Reply
  11. Viacheslav said:

    Hello,
    Can I ask you one question
    Explain me in details, whence you take result_flag?
    If you can, show the result_flag in code

    August 10, 2017
    Reply
    • Shivahari P Shivahari P said:

      Hi,
      We use result_flag to validate if a particular step passed/failed.

      #Set Phone number in form
      result_flag = test_obj.set_phone(phone)
      test_obj.log_result(result_flag,
                          positive="Phone number was successfully set for phone: %sn"%phone,
                          negative="Failed to set phone number: %s nOn url: %sn"%(phone,test_obj.get_current_url()))
       #Update TestRail
       case_id = testrail_file.test_example_form_phone
       test_obj.report_to_testrail(case_id,test_run_id,result_flag)
      August 10, 2017
      Reply
  12. Sebi said:

    Thanks a lot . This really helped me.

    January 22, 2018
    Reply
  13. Kranthi said:

    Hi,
    Can you pls help me in getting the multiple testcases using multiple test suite ids. Using the below snippet I can get testcases that are releated to the specific testsuite
    from testrail import *
    client = APIClient(‘https://XXXX’)
    client.user = ‘…..’
    client.password = ‘….’
    case = client.send_get(‘get_cases/1&suite_id=3881’)
    print(case)

    June 7, 2018
    Reply
  14. Kranthi said:

    Need help in fetching testcases using multiple testsuite id
    from testrail import *
    client = APIClient(‘…..’)
    client.user = ‘…..’
    client.password = ‘….’
    case = client.send_get(‘get_cases/1&suite_id=xxx’)
    print(case)

    June 7, 2018
    Reply
    • Indira Nellutla Indira Nellutla said:

      Kranthi, I looked at the TestRail API docs also and couldn’t find any details about using multiple suite id’s. Will surely let you know if I find any information.

      Thanks
      Indira Nellutla

      June 8, 2018
      Reply
  15. Daniel said:

    Hi, I have a Framework using Behave (gherkin), where every scenarios is a test case.

    How can I update a testrail run having this sintax? Where should I put the test case id?

    Thanks

    June 18, 2018
    Reply
    • Rohan Dudam Rohan Dudam said:

      Hi Daniel,
      Looks like you need to use “Behave to TestRail Reporter” package for integration. Please refer the following link for more details about how to setup and use: https://pypi.org/project/behave-testrail-reporter/

      Thanks,
      Rohan Dudam

      June 21, 2018
      Reply
      • Daniel said:

        Thank you Rohan, I did check behave-testrail-reporter but it was difficult to understand, instead I used the example from this page with minor changes, and to use it with behave was really easy:

        I create a step like: Then I will run the following test case: “439”
        That step saves into the context variable (special global variable from behave) the tc_id, then using the test_rail.py I update the testplan with the tc_result + tc_id at the after_scenario

        🙂

        August 29, 2018
        Reply
  16. silfredo mora said:

    ty very detailed test I used a different approach but this was great a example, ty very much

    October 10, 2018
    Reply
  17. rin said:

    Hi
    I have a jenkins job to run CI automation tests and i want to report results back to Test rail how can i do this? please help

    September 16, 2019
    Reply
    • Rohan Dudam Rohan Dudam said:

      Hi,
      Here you just need to modify your test to post a report to TestRail as per blog. CI (Jenkins) is only responsible for triggering your test. If your test capable to post report to TestRail, then you don’t worry about CI.

      September 17, 2019
      Reply
  18. Anonymous said:

    Hi How can we add(Push ) a defect to JIRA via TestRail using python.

    December 11, 2019
    Reply
  19. Namita said:

    How can we Push a Defect to JIRA using the defect plugin in python.

    December 11, 2019
    Reply
  20. Shilpa said:

    how to pass custom fields via pytest automation and post in testrail.
    I have purest-testrail plugin integrated with purest framework. When I. run automation tests to post. in test rail, the test results are posted with Untested as the test run status.
    This is because field Platform is not passed.
    How can I pass this custom. field “Platform” via automation so that the tes trun status is updated correctly

    April 20, 2020
    Reply

Leave a Reply

Your email address will not be published.