{"id":6118,"date":"2017-06-14T01:30:49","date_gmt":"2017-06-14T05:30:49","guid":{"rendered":"https:\/\/qxf2.com\/blog\/?p=6118"},"modified":"2018-04-02T10:35:22","modified_gmt":"2018-04-02T14:35:22","slug":"post-pytest-test-results-on-slack","status":"publish","type":"post","link":"https:\/\/qxf2.com\/blog\/post-pytest-test-results-on-slack\/","title":{"rendered":"Post pytest test results on Slack"},"content":{"rendered":"

Slack<\/a> is a really popular instant messenger among our clients. It has very nice integration capabilities (Slack bots) that we use with our CI setup. At Qxf2<\/a>, we sometimes encourage (and enable!) developers to run automated GUI and API tests against their local builds. So, we decided to enhance our GUI automation framework to post the test result of a local run on Slack<\/a>. Our framework uses pytest<\/a> as our test runner. This post shows you how to post pytest’s test results on Slack.<\/p>\n

\n

Steps to post the test reports on the Slack:<\/h3>\n

To post reports on the Slack channel, you need to follow the below 5 steps:<\/p>\n

    \n
  1. Setup Slack Incoming WebHooks<\/li>\n
  2. Write script to post the test report\/any message on the Slack channel<\/li>\n
  3. Use pytest\u00a0plugin hook to call Slack integration<\/li>\n
  4. Put it all together<\/li>\n
  5. Run the test<\/li>\n<\/ol>\n
    \n

    Step-1 Setup Slack Incoming WebHooks:<\/h3>\n

    To generate Slack incoming webhook URL, you need to do following steps:<\/p>\n

      \n
    1. With help of the Slack link available here<\/a>, login and navigate to custom integration option shown in Fig. 1.\n

      \"\"<\/a>
      Fig. 1 Custom Integration Options<\/figcaption><\/figure><\/li>\n
    2. \u00a0From custom integration option list, select ‘Incoming WebHooks’ option and select a channel or create a new channel to post the test reports\/message.\n

      \"\"<\/a>
      Fig.2 Choose a channel or create a new channel to post the test reports<\/figcaption><\/figure><\/li>\n
    3. After choosing a channel, click on ‘Add Incoming WebHooks integration’ Button which will generate Incoming WebHook URL as shown in Fig 3. And also allow a user to edit details and add the custom icon.\n

      \"\"<\/a>
      Fig. 3 Generated Incoming WebHook URL<\/figcaption><\/figure><\/li>\n
    4. Now copy\/note down the generated Incoming WebHook URL and save settings. Mostly Incoming WebHook URL has the following format: \u00a0https:\/\/hooks.slack.com\/services\/TXXXXXXXX\/BXXXXXXXX\/XXXXXXXXXXXXXXXXXXXXXXX \u00a0We need this URL to post the test reports\/any message on Slack channel.<\/li>\n<\/ol>\n
      \n

      Step-2 Write script to post the test report on Slack:<\/h3>\n

      To post the test report or any message on the Slack channel, we require slack incoming webhook URL which we generated in the previous step and a test report in .txt or .log format. To generate a test report in .txt or .log format, we need to use > test_report.log<\/code> argument along with the pytest\u00a0command:<\/p>\n

      py.test -v > pytest_report.log<\/pre>\n
      Once you get both slack incoming webhook and test report in .log or .txt format, you can post the test report on the slack channel using the following script\/code.<\/p>\n
      #Contents of post_test_reports_to_slack.py \r\nimport json,os,requests\r\n\r\ndef post_reports_to_slack():\r\n        \r\n        url= \"Put your Slack incoming webhook url here\"  \r\n    \r\n        #To generate report file add \"> pytest_report.log\" at end of py.test command for e.g. py.test -v > pytest_report.log\r\n        test_report_file = os.path.abspath(os.path.join(os.path.dirname(__file__),'pytest_report.log')) #Add report file name and address here\r\n        \r\n        # Open report file and read data\r\n        with open(test_report_file, \"r\") as in_file:\r\n                testdata = \"\"\r\n                for line in in_file:\r\n                        testdata = testdata + '\\n' + line\r\n                \r\n        # Set Slack Pass Fail bar indicator color according to test results   \r\n        if 'FAILED' in testdata:\r\n            bar_color = \"#ff0000\"\r\n        else:\r\n            bar_color = \"#36a64f\"\r\n \r\n        # Arrange your data in pre-defined format. Test your data format here: https:\/\/api.slack.com\/docs\/messages\/builder?  \r\n        data = {\"attachments\":[\r\n                            {\"color\": bar_color,\r\n                            \"title\": \"Test Report\",\r\n                            \"text\": testdata}\r\n                            ]}\r\n        json_params_encoded = json.dumps(data)\r\n        slack_response = requests.post(url=url,data=json_params_encoded,headers={\"Content-type\":\"application\/json\"})\r\n        if slack_response.text == 'ok':\r\n                print '\\n Successfully posted pytest report on Slack channel'\r\n        else:\r\n                print '\\n Something went wrong. Unable to post pytest report on Slack channel. Slack Response:', slack_response \r\n        \r\n#---USAGE EXAMPLES\r\nif __name__=='__main__':\r\n        post_reports_to_slack()\r\n<\/pre>\n
      \n
      Step-3 Use pytest\u00a0plugin hook to call Slack integration:<\/h3>\n
      To get the status of all tests built in a session, we need to call slack integration script after completion of the entire test run. In pytest<\/a>, there is a provision to run scripts\/code after execution of all tests. pytest_sessionfinish<\/code> plugin hook executes after the whole test run finishes. So we used pytest_sessionfinish<\/code>\u00a0plugin hook to call Slack integration script. To know more about the pytest\u00a0plugin hook refer to _pytest.hookspec\u00a0doc<\/a>.<\/p>\n
      To use pytest_sessionfinish<\/code> plugin hook, we need to modify conftest.py<\/code> with the following code:<\/p>\n
      #Contents of conftest.py\r\nimport post_test_reports_to_slack # import slack integration file\r\n\r\n#Test arguments\r\n@pytest.fixture\r\ndef slack_integration_flag():\r\n    \"pytest fixture for os version\"\r\n    return pytest.config.getoption(\"-I\")\r\n\r\n#Command line options:\r\ndef pytest_addoption(parser):\r\n    \"add parser options\"\r\n    parser.addoption(\"-I\",\"--slack_integration_flag\",\r\n                      dest=\"slack_integration_flag\",\r\n                      default=\"N\",\r\n                      help=\"Post the test report on slack channel: Y or N\")\r\n#pytest plugin hook\r\ndef pytest_sessionfinish(session, exitstatus):\r\n    \"executes after whole test run finishes.\"\r\n    if pytest.config.getoption(\"-I\").lower() == 'y':\r\n        post_test_reports_to_slack.post_reports_to_slack()\r\n<\/pre>\n
      Note: In above code, we added a slack integration flag command line option to enable\/disable slack integration.<\/p>\n
      \n
      Step-4 Put it all together:<\/h3>\n
      We are including a sample test for you to check out the pytest and Slack integration. The test script looks like the code below:<\/p>\n
      #content of test_example_form.py  \r\n\"\"\"\r\nQxf2 Services: Utility script to test example form\r\nNOTE: This is a contrived example that was written up to make this blog post clear\r\nWe do not use this coding pattern at our clients\r\n\"\"\"\r\n \r\nfrom selenium import webdriver\r\nimport sys,time\r\n \r\ndef test_example_form(browser):\r\n    \"Test example form\"\r\n \r\n    #Create an instance of WebDriver\r\n    if browser.lower() == 'firefox':\r\n            driver = webdriver.Firefox()\r\n    elif browser.lower() == 'chrome':\r\n            driver = webdriver.Chrome()\r\n \r\n    #Create variables to keep count of pass\/fail\r\n    pass_check_counter = 0\r\n    total_checks = 0\r\n \r\n    #Visit the tutorial page\r\n    driver.get('http:\/\/qxf2.com\/selenium-tutorial-main') \r\n    #Check 1: Is the page title correct?\r\n    if(driver.title==\"Qxf2 Services: Selenium training main\"):\r\n        print (\"Success: Title of the Qxf2 Tutorial page is correct\")\r\n        pass_check_counter += 1\r\n    else:\r\n        print (\"Failed: Qxf2 Tutorial page Title is incorrect\")\r\n    total_checks += 1\r\n \r\n    #Fill name, email and phone in the example form\r\n    name_field = driver.find_element_by_xpath(\"\/\/input[@type='name']\")\r\n    name_field.send_keys('Rohan')\r\n    email_field = driver.find_element_by_xpath(\"\/\/input[@type='email']\")\r\n    email_field.send_keys('test@qxf2.com')\r\n    phone_field = driver.find_element_by_xpath(\"\/\/input[@type='phone']\")\r\n    phone_field.send_keys('9999999999')\r\n    submit_button = driver.find_element_by_xpath(\"\/\/button[@type='submit']\") #Click on the Click me button\r\n    submit_button.click()\r\n    time.sleep(5)\r\n    #Check 2: Is the page title correct?\r\n    if(driver.title==\"Qxf2 Services: Selenium training redirect\"):\r\n        print (\"Success: The example form was submitted\")\r\n        pass_check_counter += 1\r\n    else:\r\n        print (\"Failed: The example form was not submitted. Automation is not on the redirect page\")\r\n    total_checks += 1\r\n    #Quit the browser window\r\n    driver.quit() \r\n \r\n    #Assert if the pass and fail check counters are equal\r\n    assert total_checks == pass_check_counter \r\n \r\n#---START OF SCRIPT\r\nif __name__=='__main__':\r\n    browser = sys.argv[1] #Note:using sys.argv to keep this example short. We use OptionParser with all our clients \r\n    text_example_form(browser)\r\n<\/pre>\n
      The completed conftest.py<\/code> looks like this:<\/p>\n
      #Contents of conftest.py\r\nimport pytest\r\nimport os\r\nimport post_test_reports_to_slack # import slack integration file\r\n\r\n#Test arguments\r\n@pytest.fixture\r\ndef browser():\r\n    \"pytest fixture for browser\"\r\n    return pytest.config.getoption(\"-B\")\r\n\r\n@pytest.fixture\r\ndef slack_integration_flag():\r\n    \"pytest fixture for os version\"\r\n    return pytest.config.getoption(\"-I\")\r\n\r\n#command line options\r\ndef pytest_addoption(parser):\r\n    parser.addoption(\"-B\",\"--browser\",\r\n                      dest=\"browser\",\r\n                      default=\"firefox\",\r\n                      help=\"Browser. Valid options are firefox, ie and chrome\")\r\n\r\n    parser.addoption(\"-I\",\"--slack_integration_flag\",\r\n                      dest=\"slack_integration_flag\",\r\n                      default=\"N\",\r\n                      help=\"Post the test report on slack channel: Y or N\")\r\n\r\n#pytest plugin hook\r\ndef pytest_sessionfinish(session, exitstatus):\r\n    \"executes after whole test run finishes.\"\r\n    if pytest.config.getoption(\"-I\").lower() == 'y':\r\n        post_test_reports_to_slack.post_reports_to_slack()\r\n<\/pre>\n
      \n
      Step-5 Run the test:<\/h3>\n
      To run the test use the command py.test -v -I Y > pytest_report.log<\/code><\/p>\n
      After completion of the test, you receive test report on Slack channel. Look at Fig 4 to see a screenshot of test report notification we received on Slack.<\/p>\n
      \"\"<\/a>
      Fig. 4: Test Report on Slack channel<\/figcaption><\/figure>\n
      \n
      Special Case: xdist and Slack integration<\/h3>\n
       Recently, we come across an issue with use of pytest_sessionfinish<\/code> plugin hook. When we ran our tests in parallel (using pytest-xdist<\/a>), we received multiple messages on the Slack channel. This is because when we run tests in parallel, tests get distributed across multiple CPU\/subprocesses which call pytest_sessionfinish<\/code> plugin hook multiple times.
      \nTo solve this problem, we used pytest_terminal_summary<\/code> plugin hook instead of using pytest_sessionfinish<\/code> plugin hook. To use pytest_terminal_summary<\/code> plugin hook you need to small modification in conftest.py<\/code>. You need to replace pytest_sessionfinish<\/code> method with pytest_terminal_summary<\/code> method given below.<\/p>\n
      #Replace complete pytest_sessionfinish method with following method\r\ndef pytest_terminal_summary(terminalreporter, exitstatus):\r\n    \"add additional section in terminal summary reporting.\"\r\n    if pytest.config.getoption(\"-I\").lower() == 'y':\r\n        post_test_reports_to_slack.post_reports_to_slack()           \r\n<\/pre>\n
      If you are a startup finding it hard to hire technical QA engineers, learn more about Qxf2 Services<\/a>.<\/strong><\/p>\n
      \n