We, at Qxf2, are really liking pytest. It is the most Pythonic test runner we have come across so far. We started using it with our GUI automation. We realized one thing very quickly – pytest’s reporting capabilities are configured to suit unit testing. We needed to reconfigure pytest’s reporting to suit our GUI automation framework. It was also hard to Google for the exact changes we wanted to do. So, once we figured out what to do, we decided to write a post showing you how to modify and control the different parts of pytest’s failure reports.

Setup

We realize that no single configuration of output message suits everybody. So we decided to provide you with a simple but comprehensive sample test. The sample will help you follow along with this post. And if you do not like how we tweaked pytest’s failure reporting, you can use the sample test to try out other configurations.

Our sample test has the following features:
a) writes out to stdout (stream handler)
b) writes out to a log file (file handler)
c) we force the assert fail so we can show a failure summary
d) we have a list of meaningful failure messages that we would like to display
e) we have some test metadata (e.g.: browser) we want displayed

Here is the sample test code (file name is test_pytest_summary.py):

"""
A sample test writing out the error to the log and printing the failure message as the output.
"""
#START HERE
def test_my_output(): #1. Defining the method
 
      #2. Debug/progress messages
      import logging
      log = logging.getLogger('file_log') #A log handler
      fileHandler = logging.FileHandler('file_log.log')
      log.addHandler(fileHandler)
      streamHandler = logging.StreamHandler() #Write out to the command prompt
      log.addHandler(streamHandler)
 
      log.error('Hello from log: You should not see me in the pytest summary') #write into a log
      print 'Hello from print: You should not see me in the pytest summary' #
 
      #3. results summary section
      log.error('Hello from error: This is a summary of an error')
      print 'Hello from not-an-error: This is a summary of stdout aka not-an-error'
 
      #4. Example list of failures that your test may have collected
      test_metadata = 'firefox 45, OS X Yosemite'
      failure_list = ['1. Ineffective kiss. Frog did not turn into a prince.','2. The Queen used rat poison in the apple.','3. The birds   ate the breadcrumbs.']
      assert 3 == 4
 
#---START OF SCRIPT
if __name__=='__main__':
      #5. call the method
      test_my_output()

The pytest failure report

Run the above test code by using the py.test test_pytest_summary.py. You will notice that the failure summary looks like the image below:

The pytest failure report has three parts:
a) the failure/traceback section
b) the captured stdout section
c) the captured stderr section

The failure report is optimized for unit testing. For example, it shows the entire method that failed as part of the traceback. That is/may be a good approach for a unit test but is not very useful for a broadstack test with multiple checkpoints. We would rather have the GUI/broadstack test go as far along as possible while collecting a list of failures and then displaying all the failures at the end. We also write numerous log messages to the console as part of our GUI automation. We would rather not have our entire verbose log displayed as part of the failure report.

NOTE: We are showing you how to suppress a stderr message too. We do not really know if this will ever be useful – but we thought for the sake of completeness, we should show you how to control that section too.

An improved pytest failure report

To get a better failure summary, you need to do the following:
1. Stop the failure/traceback section from displaying the entire method
2. Add our human-friendly failure messages to the failure section
3. Flush the stdout/stderr buffers and insert only the messages we want

1. Stop the failure/traceback section from displaying the entire method

This is easy. pytest provides a --tb command line option to control the traceback section. We preferred the output when --tb was set to short. Just run the above test code, use py.test test_pytest_summary.py --tb=short and notice that the entire method is no longer displayed as part of the failure/traceback section.

2. Add our human-friendly failure messages to the failure section

To add human-friendly failure messages, simply add a comma after the assert statement and add the human-friendly messages. In our example, it will look like the snippet below:

assert 3 == 4, "\n----TEST META DATA----\n%s\n----FAILURE-LIST----\n%s"%(test_metadata,'\n'.join(failure_list))

If you run the test using the command py.test test_pytest_summary.py --tb=short, the output will look something like this:

Notice that the failure summary is formatted, the details of the test method do not appear in the traceback and the human-friendly failure messages have been included as part of the failure section.

3. Flush the stdout/stderr buffers and insert only the messages we want

To control the captured stdout/stderr sections, we need to use pytest’s capsys fixture. The capsys.readouterr() call snapshots the output so far. After the test function finishes, the original streams will be restored. Using the capsys fixture this way frees your test from having to care about setting/resetting output streams and also interacts well with pytest’s own per-test capturing.

Modify the test as follows:
a. Add an argument called capsys to the test function.

def test_my_output(capsys)

b. Add these lines of code to the method wherever you want to flush out the stdout, stderr buffers

#3. The key lines of code
      if capsys is not None:
            out,err = capsys.readouterr() #Flushes out the stdout, stderr buffers

This way, pytest will only output the stdout and stderr messages that were written after the buffers were flushed. If you do not want to lose the stderr messages, simply log err back as errors.

NOTE: Just as we finished writing this blog post, pytest came out with a with capsys.disabled() option. We have not had enough time to experiment with this option yet. So, to learn more, check out their official post here.

Putting it all together

Now our test code (test_pytest_summary.py) should look like this:

"""
Qxf2 Services:
A sample test writing out the error to the log and printing the failure message as the output.
This is a contrived example to help readers follow along with out blog post 
"""
def test_my_output(capsys): #1. Defining the method with the argument capsys
      "Contrived test method to serve as an illustrative example"
      #2. Debug/progress messages
      import logging
      log = logging.getLogger('file_log') #A log handler
      fileHandler = logging.FileHandler('file_log.log')
      log.addHandler(fileHandler)
      streamHandler = logging.StreamHandler() #Write out to the command prompt
      log.addHandler(streamHandler)
 
      log.error('Hello from log: You should not see me in the pytest summary') #write into a log
      print 'Hello from print: You should not see me in the pytest summary' #
 
      #3. The key lines of code
      if capsys is not None:
            out,err = capsys.readouterr() #Flushes out the stdout, stderr buffers
 
      #4. Results summary section
      log.error('Hello from error: This is a summary of an error')
      print 'Hello from not-an-error: This is a summary of stdout aka not-an-error'
 
      #5a. Example test meta-data
      test_metadata = 'firefox 45, OS X Yosemite'
      #5b. Example list of failures that your test may have collected
      failure_list = ['1. Ineffective kiss. Frog did not turn into a prince.','2. The Queen used rat poison in the apple.','3. The birds ate the breadcrumbs.']
      assert 3 == 4, "\n----TEST META DATA----\n%s\n----FAILURE-LIST----\n%s"%(test_metadata,'\n'.join(failure_list))
 
#---START OF SCRIPT
if __name__=='__main__':
      #6. call the method & make capsys the first argument set it None by default
      test_my_output(capsys=None)

Run the test

Run the test using the command py.test test_pytest_summary.py --tb=short where, -- tb=short is the flag for shorter traceback format.

W00t! The pytest failure report is so much nicer now.

And this is how we ended up re-configuring pytest’s failure summary to suit our GUI automation needs. If you have questions, please post them below and one of us will get back to you soon.

If you liked what you read, know more about Qxf2.

Related posts:

How to investigate pytest-warnings Exploring pytest command line options Pytest, argparse.ArgumentError and the -V: conflicting option