Problem: How do you verify what was drawn on a canvas element using Selenium?
Why this post?
The HTML5 canvas element is opaque to Selenium and does not lend itself well to GUI automation. An application I was testing had an HTML5 canvas element that users could interact with. Users could place pre-defined objects upon the canvas element and then print out the canvas. This area of the software repeatedly had regressions and needed constant attention from testers. I wrote an automated check that, while not ideal, was a big improvement over the existing situation.
I use an indirect way to verify what is drawn on an HTML5 canvas. The approach relies on having baseline images of how the canvas is expected to look like at each step of the drawing. At a high level we will:
1. Use the canvas element’s toDataURL() method to get an image of the canvas
3. Compare the image generated with a baseline
Step 1. The canvas element’s toDataURL() method
Let’s begin by looking at the builtin toDataURL() method of all HTML5 canvas elements.
The HTMLCanvasElement.toDataURL() method returns a data URIs containing a representation of the image in the format specified by the type parameter (defaults to PNG). Source:The docs
If you cannot figure this out on your own, go ahead and request your developers for help.
Step 2. Selenium’s execute_script() method
# Get the image png_url = self.driver.execute_script('return document.getElementsByClassName("my-canvas").toDataURL("image/png");')
Next, you will need to convert the long string that the toDataURL() method returns and save it as a PNG.
#Parse the URI to get only the base64 part str_base64 = re.search(r'base64,(.*)',png_url).group(1) #Convert it to binary str_decoded = str_base64.decode('base64') #Write out the image somewhere output_img = "you fill the path here like /tmp/checkpoint1.png" fp = open(output_img,'wb') fp.write(str_decoded) fp.close()
BOOM! You just exported the current canvas as an image.
Step 3. Image compare
The next step is to compare the image with a known baseline. I am using Python and its Python Image Library (PIL). You can try something similar in other languages too. Here is a code snippet showing how to compare two images.
from PIL import Image, ImageChops import os def is_equal(img_actual,img_expected,result): "Returns true if the images are identical(all pixels in the difference image are zero)" result_flag = False #Check that img_actual exists if not os.path.exists(img_actual): print 'Could not locate the generated image: %s'%img_actual #Check that img_expected exists if not os.path.exists(img_expected): print 'Could not locate the baseline image: %s'%img_expected if os.path.exists(img_actual) and os.path.exists(img_expected): actual = Image.open(img_actual) expected = Image.open(img_expected) result_image = ImageChops.difference(actual,expected) #Where the real magic happens if (ImageChops.difference(actual,expected).getbbox() is None): result_flag = True #Bonus code to store the overlay #Result image will look black in places where the two images match color_matrix = ( + ( * 255)) result_image = result_image.convert('L') result_image = result_image.point(color_matrix) result_image.save(result)#Save the result image return result_flag
Now you can generate images every time you draw something on the canvas and compare them with pre-defined baselines.
Caution: The images you produce depend on your screen resolution. So there is some housekeeping involved in making the test repeatable across all resolutions.
Voila! You now have an automated way to check what was drawn on an HTML5 canvas. I will follow up soon with a post on how to use Selenium to handle different screen resolutions, drag & drop, drag & click and identify coordinates on a canvas element. Stay tuned!
PS: Looking for Python-based web automation framework? Try our open-sourced GUI automation framework based on the page object model.