This post will map to our experience. It will serve as a short guide to help you go from a beginner at Accessibility testing to integrating Axe with your automated tests.<\/p>\n
Part I: Pick a tool<\/h3>\n
In our exploration of accessibility tools for accessibility testing of both internal projects and at client end, we initially looked up for a suitable candidate for accessibility testing. We noticed that many of the tools like wave<\/a> had one thing in common that they relied on using web URLs and lacked integration capabilities with automation suite which was a key requirement for us.<\/p>\n Our search for alternative option for accessibility led us to Axe<\/a>, a renowned tool in the accessibility testing domain. What stood out about Axe<\/a> was its capacity to seamlessly integrate with automation suites, making it a good choice for our accessibility testing needs. This also prompted us to delve deeper into Axe’s functionalities, intrigued by its potential to cater to both technical and non-technical users. We will now focus more into detailed exploration to understand how Axe precisely addresses our accessibility testing requirements.<\/p>\n While exploring Axe<\/a> tool, we found that there were two ways how we can use Axe as a accessibility tool. One was through browser plugin and another was by integrating it to our automation suite. We tried both the way.<\/p>\n axe DevTools is the browser extension to test the web application. It can be easily plugged with the browser. It can be installed from web store and than once installed will be available in the developers tool.<\/p>\n Once the axe DevTools is installed, we navigate to the web application to uncover the accessibility issues and open the developer tools. We click the axe Devtools which gives us two options: Intelligent guided test is a more advanced way of doing accessibility testing where using a simple question and answer format about the page and content under test, Intelligent guided test leverages machine learning to help you quickly find and fix issues that are not detectable by automated testing alone. In this section we will show you how to integrate Axe with your test automation framework. We will focus on integrating Axe with Qxf2’s test automation framework<\/a>. But the steps remain similar for any framework of your choice.<\/p>\n axe-core is the accessibility engine for automated Web UI testing and we will be using this as a part of our automation suite moving forward. Let’s see how we can use it.<\/p>\n Installing axe-core is straight forward. To install Axe with selenium, use the pip package manager.<\/p>\n Axe-core has different types of rules, for WCAG 2.0, 2.1, 2.2 on level A, AA and AAA along with best practices to identify common accessibility practices like ensuring every page have a h1 heading.<\/p>\n The axe-core API<\/a> consists of: As we started the integration, we thought it would be straight forward where we need to inject the JavaScript and run it on the pages and get the accessibility issues but there were few issues we hit along the way. We will discuss them, so that anyone facing this issue doesn’t spend much time debugging.<\/p>\n In general there are two methods in Axe viz. inject and run which are often shown in online examples but when integrating within a framework we have to do a bit more than just using the methods directly. We dug a level deep and checked the constructor of the Axe class and realized that we were missing an important param called script_url<\/strong>. script_url will point to the file containing the axe.min.js which would be injected in the page. We had to put the axe.min.js in our framework and point the script_url to it before using any of the Axe methods. Once the script_url was included, we had to call the parent class constructor and use super() method to call the methods in the parent class i.e. Axe class.<\/p>\n The methods of our interest were inject() and run() in the parent class. We happened to use a wrapper around the methods as shown below.<\/p>\n So, with this we were able to use the methods in our tests but there was a problem. When we were running the accessibility test, we were getting one test results for all the pages. That would be a mess for anyone to figure out which accessibility issue is for which page. Since, we will be dealing with multiple pages, we would need a accessibility report for each pages. To handle this, we have introduced a method in the PageFactory of our framework which would return all the pages. We than iterate over the pages in our tests, inject and run Axe and create a result for every page.<\/p>\n This is how a part of test looks like by fetching the pages from PageFactory and iterating over each page and using Axe.<\/p>\n Now as we were able to run the accessibility tests with accessibility result for every page, we were thinking how the test result will be useful because whenever there is a change in the UI, it is most likely to introduce new accessibility issue but the test will run and capture the latest result and there is no way we can compare the change with last run or know what changed, or worst even the change is acceptable or not. Looking up on this, we came across snapshots which would allow us to have a snapshot for reference and compare the result for each test run.<\/p>\n So, using snapshot was simple. We had to install a plugin for snapshot testing with pytest.<\/p>\n We created a class that calls the parent Snapshot class using the super() method.<\/p>\n To generate snapshot for the first time, use the following command-line argument while running the test script:<\/p>\n Passing the above command-line argument while running the test script creates a snapshot directory along with the snapshot file for the first time. If the snapshots already exist, it updates them with the latest copy of Axe violations in the test pages.<\/p>\n We use the ‘compare_and_log_violation()’ method to compare the current test results with existing snapshots using the DeepDiff library. If no new violations are found while comparing against the baseline snapshot, the test passes. If violations are resolved or new violations are found, they are logged in the console and in a separate file located in ‘.\/conf\/new_violations_record.txt’.<\/p>\n So, with the above methods of Snapshot class, the test script runs accessibility checks on each page, compares the results with stored snapshots, and logs any changes found.<\/p>\n Here is the complete test script of Axe accessibility testing.<\/p>\n This is how a typical Axe test result looks. As part of our Newsletter Automation <\/strong> project, we have integrated accessibility tests into our GitHub workflow. In addition to running API and UI tests as part of CI, we wanted to include accessibility tests as well. However, we debated where these tests should fit\u2014before or after functional tests? The decision largely depended on the project\u2019s priorities and requirements. While we are still evaluating the best placement, we have currently positioned them before functional tests.<\/p>\n The next problem we were getting was the frequent failure of accessibility tests. This would be a common phenomena as the web under test would evolve and there would be frequent UI changes but we don\u2019t want the CI to fail, stop and not execute the other tests because our accessibility tests failed.Narrowing down to Axe from deque<\/h5>\n
\nPart II: Starting with axe DevTools<\/h3>\n
Scanning all pages at one go to get accessibility result<\/h5>\n
\n1. Scan ALL of my page
\n2. Start an Intelligent Guided Test.
\nWe first tried the Scan ALL of my page and in a jiffy, it listed the accessibility issues.
\n![\"scan](\"https:\/\/qxf2.com\/blog\/wp-content\/uploads\/2023\/12\/scan_all_page.png\")
<\/a>
\nIt also allows to export the result in JSON, CSV, or JUnit XML format.<\/p>\nIntelligent Guided Test<\/h5>\n
\nWith Intelligent guided test, you can also make your testing more agile by focussing only on specific parts of the page like a table, images or forms.
\nWe have focussed less on Intelligent guided test because we were more interested to get Axe<\/a> integrated with automation suite and the axe DevTools extension gave us a good start.<\/p>\n
\nPart III: Integrating Axe with test automation<\/h3>\n
1. Exploring axe-core<\/h4>\n
Installation of axe-core<\/h5>\n
pip install axe-selenium-python<\/pre>\n
Content of the axe-core API<\/h5>\n
\naxe.min.js – a minified version of the java script file to be used in our framework. Axe would inject this file into all frames for evaluating the contents inside that frame. This is the file which would help identify the problems related to accessibility.<\/p>\n2. Integration of axe-core with Qxf2 framework<\/h4>\n
Include axe.min.js in your code base<\/h5>\n
import os\r\nfrom axe_selenium_python import Axe\r\n\r\nscript_url=os.path.abspath(os.path.join(os.path.dirname(__file__), \"..\", \"utils\", \"axe.min.js\"))\r\n\r\nclass Accessibilityutil(Axe):\r\n \"Accessibility object to run accessibility test\"\r\n def __init__(self, driver):\r\n super().__init__(driver, script_url)\r\n<\/pre>\n
Write wrappers for the inject and run methods<\/h5>\n
def accessibility_inject_axe(self):\r\n \"Inject Axe into the Page\"\r\n try:\r\n return self.axe_util.inject()\r\n except Exception as e:\r\n self.write(e)\r\n\r\ndef accessibility_run_axe(self):\r\n \"Run Axe into the Page\"\r\n try:\r\n return self.axe_util.run()\r\n except Exception as e:\r\n self.write(e)\r\n<\/pre>\n
@staticmethod\r\ndef get_all_page_names():\r\n \"Return the page names\"\r\n return [\"main page\",\r\n \"redirect\",\r\n \"contact page\"]\r\n<\/pre>\n
Example tests<\/h5>\n
@pytest.mark.ACCESSIBILITY\r\ndef test_accessibility(test_obj):\r\n \"Inject Axe and create snapshot for every page\"\r\n try:\r\n\r\n #Initalize flags for tests summary\r\n expected_pass = 0\r\n actual_pass = -1\r\n\r\n #Get all pages\r\n page_names = PageFactory.get_all_page_names()\r\n\r\n for page in page_names:\r\n test_obj = PageFactory.get_page_object(page,base_url=test_obj.base_url)\r\n #Inject Axe in every page\r\n test_obj.accessibility_inject_axe()\r\n #Check if Axe is run in every page\r\n run_result = test_obj.accessibility_run_axe()\r\n<\/pre>\n
3. Using snapshot testing against Axe result<\/h4>\n
pip install pytest-snapshot<\/pre>\n
import conf.snapshot_dir_conf\r\nfrom pytest_snapshot.plugin import Snapshot\r\n\r\nsnapshot_dir = conf.snapshot_dir_conf.snapshot_dir\r\n\r\nclass Snapshotutil(Snapshot):\r\n \"Snapshot object to use snapshot for comparisions\"\r\n def __init__(self, snapshot_update=False,\r\n allow_snapshot_deletion=False,\r\n snapshot_dir=snapshot_dir):\r\n super().__init__(snapshot_update, allow_snapshot_deletion, snapshot_dir)\r\n<\/pre>\n
python -m pytest tests\/test_accessibility.py --snapshot-update<\/pre>\n
Using DeepDiff library for snapshot comparisons<\/h5>\n
# Use deepdiff to compare the snapshots\r\nviolation_diff = DeepDiff(existing_snapshot_dict,\r\n current_violations_dict,\r\n ignore_order=True,\r\n verbose_level=2)\r\n<\/pre>\n
@pytest.mark.ACCESSIBILITY\r\ndef test_accessibility(test_obj, request):\r\n \"Test accessibility using Axe and compare snapshot results and save if new violations found\"\r\n try:\r\n\r\n #Initalize flags for tests summary\r\n expected_pass = 0\r\n actual_pass = -1\r\n\r\n #Get snapshot update flag from pytest options\r\n snapshot_update = request.config.getoption(\"--snapshot_update\")\r\n #Create an instance of Snapshotutil\r\n snapshot_util = Snapshotutil(snapshot_update=snapshot_update)\r\n\r\n #Set up the violations log file\r\n violations_log_path = snapshot_util.initialize_violations_log()\r\n snapshot_dir = conf.snapshot_dir_conf.snapshot_dir\r\n\r\n #Get all pages\r\n page_names = conf.snapshot_dir_conf.page_names\r\n for page in page_names:\r\n test_obj = PageFactory.get_page_object(page,base_url=test_obj.base_url)\r\n #Inject Axe in every page\r\n test_obj.accessibility_inject_axe()\r\n #Check if Axe is run in every page\r\n axe_result = test_obj.accessibility_run_axe()\r\n #Extract the 'violations' section from the Axe result\r\n current_violations = axe_result.get('violations', [])\r\n # Log if no violations are found\r\n if not current_violations:\r\n test_obj.log_result(\r\n True,\r\n positive=f\"No accessibility violations found on {page}.\",\r\n negative=\"\",\r\n level='info'\r\n )\r\n\r\n #Load the existing snapshot for the current page (if available)\r\n existing_snapshot = snapshot_util.initialize_snapshot(snapshot_dir, page, current_violations=current_violations)\r\n if existing_snapshot is None:\r\n test_obj.log_result(\r\n True,\r\n positive=(\r\n f\"No existing snapshot was found for {page} page. \"\r\n \"A new snapshot has been created in ..\/conf\/snapshot dir. \"\r\n \"Please review the snapshot for violations before running the test again. \"\r\n ),\r\n negative=\"\",\r\n level='info'\r\n )\r\n continue\r\n\r\n #Compare the current violations with the existing snapshot to find any new violations\r\n snapshots_match, new_violation_details = snapshot_util.compare_and_log_violation(\r\n current_violations, existing_snapshot, page, violations_log_path\r\n )\r\n #For each new violation, log few details to the output display\r\n if new_violation_details:\r\n snapshot_util.log_new_violations(new_violation_details)\r\n #Log the result of the comparison (pass or fail) for the current page\r\n test_obj.log_result(snapshots_match,\r\n positive=f'Accessibility checks for {page} passed',\r\n negative=f'Accessibility checks for {page} failed',\r\n level='debug')\r\n\r\n #Print out the result\r\n test_obj.write_test_summary()\r\n expected_pass = test_obj.result_counter\r\n actual_pass = test_obj.pass_counter\r\n\r\n except Exception as e:\r\n test_obj.log_result(\r\n False,\r\n positive=\"\",\r\n negative=f\"Exception when trying to run test: {__file__}\\nPython says: {str(e)}\",\r\n level='error'\r\n )\r\n\r\n assert expected_pass == actual_pass, f\"Test failed: {__file__}\"\r\n<\/pre>\n
\n![\"\"](\"https:\/\/qxf2.com\/blog\/wp-content\/uploads\/2024\/01\/axe_test_run-1024x259.jpg\")
\n<\/a>
\n <\/p>\n
\nPart IV: Running Accessibility test as a part of CI<\/h3>\n
\nSo, we used a parameter which would still allow us to execute the next tests after accessibility if it still fails. Below is the snippet of how we used if: always() in UI tests which would run after accessibility tests even if the accessibility tests fail.<\/p>\n- name: Run Accessibility Tests\r\n run: |\r\n cd tests\/integration\/tests\/accessibility_tests\r\n python -m pytest test_accessibility.py --browser headless-chrome --app_url http:\/\/localhost:5000\r\n\r\n- name: Run UI Tests\r\n if: always()\r\n run: |\r\n cd tests\/integration\/tests\/ui_tests\r\n python -m pytest -n 4 --browser headless-chrome --app_url http:\/\/localhost:5000\r\n<\/pre>\n