Problem: How do you verify what was drawn on a canvas element using Selenium?

Why this post?

The HTML5 canvas element is opaque to Selenium and does not lend itself well to GUI automation. An application I was testing had an HTML5 canvas element that users could interact with. Users could place pre-defined objects upon the canvas element and then print out the canvas. This area of the software repeatedly had regressions and needed constant attention from testers. I wrote an automated check that, while not ideal, was a big improvement over the existing situation.

Overview:

I use an indirect way to verify what is drawn on an HTML5 canvas. The approach relies on having baseline images of how the canvas is expected to look like at each step of the drawing. At a high level we will:
1. Use the canvas element’s toDataURL() method to get an image of the canvas
2. Use Selenium’s execute_script() method to execute the toDataURL() JavaScript call
3. Compare the image generated with a baseline

Step 1. The canvas element’s toDataURL() method

Let’s begin by looking at the builtin toDataURL() method of all HTML5 canvas elements.

The HTMLCanvasElement.toDataURL() method returns a data URIs containing a representation of the image in the format specified by the type parameter (defaults to PNG). Source:The docs

Sure, this is a JavaScript call – but still useful to know. It returns an image of the canvas. So for now, identify your canvas element and get your JavaScript call ready.

return document.getElementsByClassName("my-canvas")[0].toDataURL("image/png");

If you cannot figure this out on your own, go ahead and request your developers for help.

Step 2. Selenium’s execute_script() method

Selenium has a backdoor to execute JavaScript. This is true no matter in which language you use to write your Selenium tests. Let’s use the JavaScript we wrote in Step 1 and execute it as part of our Selenium script. I’ll show you how to write the script in Python – but you can try something similar in a language of your choice.

# Get the image
png_url = self.driver.execute_script('return document.getElementsByClassName("my-canvas")[0].toDataURL("image/png");')

Next, you will need to convert the long string that the toDataURL() method returns and save it as a PNG.

#Parse the URI to get only the base64 part
str_base64 = re.search(r'base64,(.*)',png_url).group(1)
 
#Convert it to binary
str_decoded = str_base64.decode('base64')
 
#Write out the image somewhere
output_img = "you fill the path here like /tmp/checkpoint1.png"        
fp = open(output_img,'wb')
fp.write(str_decoded)
fp.close()

BOOM! You just exported the current canvas as an image.

Step 3. Image compare

The next step is to compare the image with a known baseline. I am using Python and its Python Image Library (PIL). You can try something similar in other languages too. Here is a code snippet showing how to compare two images.

from PIL import Image, ImageChops
import os
 
def is_equal(img_actual,img_expected,result):
    "Returns true if the images are identical(all pixels in the difference image are zero)"
    result_flag = False
 
    #Check that img_actual exists
    if not os.path.exists(img_actual):
        print 'Could not locate the generated image: %s'%img_actual
 
    #Check that img_expected exists
    if not os.path.exists(img_expected):
        print 'Could not locate the baseline image: %s'%img_expected
 
    if os.path.exists(img_actual) and os.path.exists(img_expected):
        actual = Image.open(img_actual)
        expected = Image.open(img_expected)
        result_image = ImageChops.difference(actual,expected)
 
        #Where the real magic happens
        if (ImageChops.difference(actual,expected).getbbox() is None):
            result_flag = True
 
        #Bonus code to store the overlay
        #Result image will look black in places where the two images match
        color_matrix = ([0] + ([255] * 255))
        result_image = result_image.convert('L')
        result_image = result_image.point(color_matrix)
        result_image.save(result)#Save the result image
 
    return result_flag

Now you can generate images every time you draw something on the canvas and compare them with pre-defined baselines.
Caution: The images you produce depend on your screen resolution. So there is some housekeeping involved in making the test repeatable across all resolutions.

Voila! You now have an automated way to check what was drawn on an HTML5 canvas. I will follow up soon with a post on how to use Selenium to handle different screen resolutions, drag & drop, drag & click and identify coordinates on a canvas element. Stay tuned!

PS: Looking for Python-based web automation framework? Try our open-sourced GUI automation framework based on the page object model.

Related posts:

Running Selenium automation using Sauce Labs: Part 3 Selenium & BrowserStack: Browsers, devices as parameters BrowserStack configuration for Selenium automation