Post pytest test results on Slack

Slack is a really popular instant messenger among our clients. It has very nice integration capabilities (Slack bots) that we use with our CI setup. At Qxf2, we sometimes encourage (and enable!) developers to run automated GUI and API tests against their local builds. So, we decided to enhance our GUI automation framework to post the test result of a local run on Slack. Our framework uses pytest as our test runner. This post shows you how to post pytest’s test results on Slack.

Steps to post the test reports on the Slack:

To post reports on the Slack channel, you need to follow the below 5 steps:

  1. Setup Slack Incoming WebHooks
  2. Write script to post the test report/any message on the Slack channel
  3. Use pytest plugin hook to call Slack integration
  4. Put it all together
  5. Run the test

Step-1 Setup Slack Incoming WebHooks:

To generate Slack incoming webhook URL, you need to do following steps:

  1. With help of the Slack link available here, login and navigate to custom integration option shown in Fig. 1.

    Fig. 1 Custom Integration Options
  2.  From custom integration option list, select ‘Incoming WebHooks’ option and select a channel or create a new channel to post the test reports/message.

    Fig.2 Choose a channel or create a new channel to post the test reports
  3. After choosing a channel, click on ‘Add Incoming WebHooks integration’ Button which will generate Incoming WebHook URL as shown in Fig 3. And also allow a user to edit details and add the custom icon.

    Fig. 3 Generated Incoming WebHook URL
  4. Now copy/note down the generated Incoming WebHook URL and save settings. Mostly Incoming WebHook URL has the following format:  https://hooks.slack.com/services/TXXXXXXXX/BXXXXXXXX/XXXXXXXXXXXXXXXXXXXXXXX  We need this URL to post the test reports/any message on Slack channel.

Step-2 Write script to post the test report on Slack:

To post the test report or any message on the Slack channel, we require slack incoming webhook URL which we generated in the previous step and a test report in .txt or .log format. To generate a test report in .txt or .log format, we need to use > test_report.log argument along with the pytest command:

py.test -v > pytest_report.log

Once you get both slack incoming webhook and test report in .log or .txt format, you can post the test report on the slack channel using the following script/code.

#Contents of post_test_reports_to_slack.py 
import json,os,requests
 
def post_reports_to_slack():
 
        url= "Put your Slack incoming webhook url here"  
 
        #To generate report file add "> pytest_report.log" at end of py.test command for e.g. py.test -v > pytest_report.log
        test_report_file = os.path.abspath(os.path.join(os.path.dirname(__file__),'pytest_report.log')) #Add report file name and address here
 
        # Open report file and read data
        with open(test_report_file, "r") as in_file:
                testdata = ""
                for line in in_file:
                        testdata = testdata + '\n' + line
 
        # Set Slack Pass Fail bar indicator color according to test results   
        if 'FAILED' in testdata:
            bar_color = "#ff0000"
        else:
            bar_color = "#36a64f"
 
        # Arrange your data in pre-defined format. Test your data format here: https://api.slack.com/docs/messages/builder?  
        data = {"attachments":[
                            {"color": bar_color,
                            "title": "Test Report",
                            "text": testdata}
                            ]}
        json_params_encoded = json.dumps(data)
        slack_response = requests.post(url=url,data=json_params_encoded,headers={"Content-type":"application/json"})
        if slack_response.text == 'ok':
                print '\n Successfully posted pytest report on Slack channel'
        else:
                print '\n Something went wrong. Unable to post pytest report on Slack channel. Slack Response:', slack_response 
 
#---USAGE EXAMPLES
if __name__=='__main__':
        post_reports_to_slack()

Step-3 Use pytest plugin hook to call Slack integration:

To get the status of all tests built in a session, we need to call slack integration script after completion of the entire test run. In pytest, there is a provision to run scripts/code after execution of all tests. pytest_sessionfinish plugin hook executes after the whole test run finishes. So we used pytest_sessionfinish plugin hook to call Slack integration script. To know more about the pytest plugin hook refer to _pytest.hookspec doc.

To use pytest_sessionfinish plugin hook, we need to modify conftest.py with the following code:

#Contents of conftest.py
import post_test_reports_to_slack # import slack integration file
 
#Test arguments
@pytest.fixture
def slack_integration_flag():
    "pytest fixture for os version"
    return pytest.config.getoption("-I")
 
#Command line options:
def pytest_addoption(parser):
    "add parser options"
    parser.addoption("-I","--slack_integration_flag",
                      dest="slack_integration_flag",
                      default="N",
                      help="Post the test report on slack channel: Y or N")
#pytest plugin hook
def pytest_sessionfinish(session, exitstatus):
    "executes after whole test run finishes."
    if pytest.config.getoption("-I").lower() == 'y':
        post_test_reports_to_slack.post_reports_to_slack()

Note: In above code, we added a slack integration flag command line option to enable/disable slack integration.

Step-4 Put it all together:

We are including a sample test for you to check out the pytest and Slack integration. The test script looks like the code below:

#content of test_example_form.py  
"""
Qxf2 Services: Utility script to test example form
NOTE: This is a contrived example that was written up to make this blog post clear
We do not use this coding pattern at our clients
"""
 
from selenium import webdriver
import sys,time
 
def test_example_form(browser):
    "Test example form"
 
    #Create an instance of WebDriver
    if browser.lower() == 'firefox':
            driver = webdriver.Firefox()
    elif browser.lower() == 'chrome':
            driver = webdriver.Chrome()
 
    #Create variables to keep count of pass/fail
    pass_check_counter = 0
    total_checks = 0
 
    #Visit the tutorial page
    driver.get('http://qxf2.com/selenium-tutorial-main') 
    #Check 1: Is the page title correct?
    if(driver.title=="Qxf2 Services: Selenium training main"):
        print ("Success: Title of the Qxf2 Tutorial page is correct")
        pass_check_counter += 1
    else:
        print ("Failed: Qxf2 Tutorial page Title is incorrect")
    total_checks += 1
 
    #Fill name, email and phone in the example form
    name_field = driver.find_element_by_xpath("//input[@type='name']")
    name_field.send_keys('Rohan')
    email_field = driver.find_element_by_xpath("//input[@type='email']")
    email_field.send_keys('[email protected]')
    phone_field = driver.find_element_by_xpath("//input[@type='phone']")
    phone_field.send_keys('9999999999')
    submit_button = driver.find_element_by_xpath("//button[@type='submit']") #Click on the Click me button
    submit_button.click()
    time.sleep(5)
    #Check 2: Is the page title correct?
    if(driver.title=="Qxf2 Services: Selenium training redirect"):
        print ("Success: The example form was submitted")
        pass_check_counter += 1
    else:
        print ("Failed: The example form was not submitted. Automation is not on the redirect page")
    total_checks += 1
    #Quit the browser window
    driver.quit() 
 
    #Assert if the pass and fail check counters are equal
    assert total_checks == pass_check_counter 
 
#---START OF SCRIPT
if __name__=='__main__':
    browser = sys.argv[1] #Note:using sys.argv to keep this example short. We use OptionParser with all our clients 
    text_example_form(browser)

The completed conftest.py looks like this:

#Contents of conftest.py
import pytest
import os
import post_test_reports_to_slack # import slack integration file
 
#Test arguments
@pytest.fixture
def browser():
    "pytest fixture for browser"
    return pytest.config.getoption("-B")
 
@pytest.fixture
def slack_integration_flag():
    "pytest fixture for os version"
    return pytest.config.getoption("-I")
 
#command line options
def pytest_addoption(parser):
    parser.addoption("-B","--browser",
                      dest="browser",
                      default="firefox",
                      help="Browser. Valid options are firefox, ie and chrome")
 
    parser.addoption("-I","--slack_integration_flag",
                      dest="slack_integration_flag",
                      default="N",
                      help="Post the test report on slack channel: Y or N")
 
#pytest plugin hook
def pytest_sessionfinish(session, exitstatus):
    "executes after whole test run finishes."
    if pytest.config.getoption("-I").lower() == 'y':
        post_test_reports_to_slack.post_reports_to_slack()

Step-5 Run the test:

To run the test use the command py.test -v -I Y > pytest_report.log

After completion of the test, you receive test report on Slack channel. Look at Fig 4 to see a screenshot of test report notification we received on Slack.

Fig. 4: Test Report on Slack channel

Special Case: xdist and Slack integration

Recently, we come across an issue with use of pytest_sessionfinish plugin hook. When we ran our tests in parallel (using pytest-xdist), we received multiple messages on the Slack channel. This is because when we run tests in parallel, tests get distributed across multiple CPU/subprocesses which call pytest_sessionfinish plugin hook multiple times.
To solve this problem, we used pytest_terminal_summary plugin hook instead of using pytest_sessionfinish plugin hook. To use pytest_terminal_summary plugin hook you need to small modification in conftest.py. You need to replace pytest_sessionfinish method with pytest_terminal_summary method given below.

#Replace complete pytest_sessionfinish method with following method
def pytest_terminal_summary(terminalreporter, exitstatus):
    "add additional section in terminal summary reporting."
    if pytest.config.getoption("-I").lower() == 'y':
        post_test_reports_to_slack.post_reports_to_slack()

If you are a startup finding it hard to hire technical QA engineers, learn more about Qxf2 Services.

Subscribe to our weekly Newsletter


View a sample


Related posts:

Default ThumbnailReportPortal integration with pytest and pytest markers Default ThumbnailPytest-xdist: Run tests in parallel Default ThumbnailEmail your pytest results as html report
Rohan Dudam

5 thoughts on “Post pytest test results on Slack”

  1. Thank you for this great article!
    From pytest 5.0 the pytest global variable became deprecated.
    Instead, it is advised to use the request.config (via the ‘request’ fixture).
    From the pytest docs: https://docs.pytest.org/en/latest/deprecations.html#pytest-config-global

    So, specifically, for our case, inside the “pytest_sessionfinish(session, exitstatus)” hook we can access the config object through session object.
    So, pytest.config.getoption(“-S”) will turn into -> session.config.getoption(“-S”)
    and consequently, the whole “slack_integration_flag” fixture can be completely removed from conftest.py(or just remove the pytest.fixture decorator).

    Reply

  2. Thanks for the great article!
    From pytest 5.0 the ‘pytest’ global variable is deprecated.
    Instead, it is advised to access the config via the request fixture (request.config). Many hooks can access the config object through different objects, indirectly
    pytest docs: https://docs.pytest.org/en/latest/deprecations.html#pytest-config-global

    In particular, in our case, inside the pytest_sessionfinish(session, exitstatus), we have to access the config object through the session object.
    So, it becomes session.config.getoption(“-I”).

    Reply

  3. This has been a great help in integrating slack reporting in the framework; however, even with using pytest_terminal_summary hook, it’s sending multiple reports to Slack at the end of the tests. Am I missing something or is there a workaround for it? My code in conftest.py looks like the following:

    def pytest_terminal_summary(terminalreporter, exitstatus):

    url = sb_config.data

    dir_path = os.path.abspath(os.path.join(os.path.dirname(__file__)))
    test_report_file = dir_path + “/tests/” + “pytest_report.log”

    with open(test_report_file, “r”) as in_file:
    testdata = “”
    for line in in_file:
    testdata = testdata + ‘\n’ + line
    test_status = “”

    # if ‘FAILURES’ or ‘ERRORS’ in testdata:
    bar_color = “#ff0000”
    test_status = “Test(s) Failed, \n see test report for details”
    else:
    bar_color = “#36a64f”
    test_status = “All Tests Passed”

    data = {“attachments”: [
    {
    “color”: bar_color,
    “title”: “Test Report for project”,
    “text”: test_status
    }
    ]}
    json_params_encoded = json.dumps(data)
    slack_response = requests.post(url=url, data=json_params_encoded,
    headers={“Content-type”: “application/json”})
    if slack_response.text == ‘ok’:
    print(‘\n Successfully posted pytest report on Slack Channel’)
    else:
    print(‘\n Something went wrong, unable to post post pytest test on Slack Channel.’, slack_response)

    Reply

    1. The code which you have mentioned is actually post_test_reports_to_slack.py, can you cross verify the code of conftest.py within the blog and check again?

      Thanks

      Reply

Leave a Reply

Your email address will not be published. Required fields are marked *