{"id":3121,"date":"2015-10-01T00:15:21","date_gmt":"2015-10-01T04:15:21","guid":{"rendered":"http:\/\/qxf2.com\/blog\/?p=3121"},"modified":"2018-03-14T10:27:53","modified_gmt":"2018-03-14T14:27:53","slug":"selenium-html5-canvas-verify-what-was-drawn","status":"publish","type":"post","link":"https:\/\/qxf2.com\/blog\/selenium-html5-canvas-verify-what-was-drawn\/","title":{"rendered":"Selenium & HTML5 canvas: Verify what was drawn"},"content":{"rendered":"

Problem:<\/strong> How do you verify what was drawn on a canvas element using Selenium?<\/p>\n

\n

Why this post?<\/h3>\n

The HTML5 canvas element is opaque to Selenium and does not lend itself well to GUI automation. An application I was testing had an HTML5 canvas element that users could interact with. Users could place pre-defined objects upon the canvas element and then print out the canvas. This area of the software repeatedly had regressions and needed constant attention from testers. I wrote an automated check that, while not ideal, was a big improvement over the existing situation. <\/p>\n

\n

Overview:<\/h3>\n

I use an indirect way to verify what is drawn on an HTML5 canvas. The approach relies on having baseline images of how the canvas is expected to look like at each step of the drawing. At a high level we will:
\n1. Use the canvas element’s toDataURL() method to get an image of the canvas
\n2. Use Selenium’s execute_script() method to execute the toDataURL() JavaScript call
\n3. Compare the image generated with a baseline <\/p>\n

\n

Step 1. The canvas element’s toDataURL() method<\/h3>\n

Let’s begin by looking at the builtin toDataURL() method of all HTML5 canvas elements.<\/p>\n

The HTMLCanvasElement.toDataURL() method returns a data URIs containing a representation of the image in the format specified by the type parameter (defaults to PNG). Source:<\/strong>The docs<\/a><\/p><\/blockquote>\n

Sure, this is a JavaScript call – but still useful to know. It returns an image of the canvas. So for now, identify your canvas element and get your JavaScript call ready.<\/p>\n

\r\nreturn document.getElementsByClassName(\"my-canvas\")[0].toDataURL(\"image\/png\");\r\n<\/pre>\n
If you cannot figure this out on your own, go ahead and request your developers for help.<\/p>\n
Step 2. Selenium’s execute_script() method<\/h3>\n
Selenium has a backdoor to execute JavaScript. This is true no matter in which language you use to write your Selenium tests. Let’s use the JavaScript we wrote in Step 1 and execute it as part of our Selenium script. I’ll show you how to write the script in Python – but you can try something similar in a language of your choice. <\/p>\n
\r\n# Get the image\r\npng_url = self.driver.execute_script('return document.getElementsByClassName(\"my-canvas\")[0].toDataURL(\"image\/png\");')\r\n<\/pre>\n
Next, you will need to convert the long string that the toDataURL() method returns and save it as a PNG. <\/p>\n
\r\n#Parse the URI to get only the base64 part\r\nstr_base64 = re.search(r'base64,(.*)',png_url).group(1)\r\n\r\n#Convert it to binary\r\nstr_decoded = str_base64.decode('base64')\r\n \r\n#Write out the image somewhere\r\noutput_img = \"you fill the path here like \/tmp\/checkpoint1.png\"        \r\nfp = open(output_img,'wb')\r\nfp.write(str_decoded)\r\nfp.close()\r\n<\/pre>\n
BOOM! You just exported the current canvas as an image.<\/p>\n
Step 3. Image compare<\/h3>\n
The next step is to compare the image with a known baseline. I am using Python and its Python Image Library (PIL). You can try something similar in other languages too. Here is a code snippet showing how to compare two images.  <\/p>\n
\r\nfrom PIL import Image, ImageChops\r\nimport os\r\n\r\ndef is_equal(img_actual,img_expected,result):\r\n    \"Returns true if the images are identical(all pixels in the difference image are zero)\"\r\n    result_flag = False\r\n\r\n    #Check that img_actual exists\r\n    if not os.path.exists(img_actual):\r\n        print 'Could not locate the generated image: %s'%img_actual\r\n   \r\n    #Check that img_expected exists\r\n    if not os.path.exists(img_expected):\r\n        print 'Could not locate the baseline image: %s'%img_expected\r\n\r\n    if os.path.exists(img_actual) and os.path.exists(img_expected):\r\n        actual = Image.open(img_actual)\r\n        expected = Image.open(img_expected)\r\n        result_image = ImageChops.difference(actual,expected)\r\n\r\n        #Where the real magic happens\r\n        if (ImageChops.difference(actual,expected).getbbox() is None):\r\n            result_flag = True\r\n\r\n        #Bonus code to store the overlay\r\n        #Result image will look black in places where the two images match\r\n        color_matrix = ([0] + ([255] * 255))\r\n        result_image = result_image.convert('L')\r\n        result_image = result_image.point(color_matrix)\r\n        result_image.save(result)#Save the result image\r\n\r\n    return result_flag\r\n<\/pre>\n
Now you can generate images every time you draw something on the canvas and compare them with pre-defined baselines.
\nCaution:<\/strong> The images you produce depend on your screen resolution. So there is some housekeeping involved in making the test repeatable across all resolutions. <\/p>\n
\n
Voila! You now have an automated way to check what was drawn on an HTML5 canvas. I will follow up soon with a post on how to use Selenium to handle different screen resolutions, drag & drop, drag & click and identify coordinates on a canvas element. Stay tuned!<\/p>\n
PS:<\/strong> Looking for Python-based web automation framework? Try our open-sourced GUI automation framework<\/a> based on the page object model. <\/p>\n
\n