At Qxf2, we have been using Infrastructure tests at our clients for a while now. We recently came up with the idea of adding some infrastructure tests for our own servers and applications within Qxf2. We used Chef Inspec to write and execute our tests. Chef InSpec is a popular open-source framework that helps you test and review your applications and infrastructure. It works by checking if your system matches the way you want it to be, which you can describe using simple and understandable Chef InSpec code. I came up with the idea for this blog when working on automating the process of running the infrastructure tests for our systems, by setting up Chef Inspec to run on Github workflow. I came across a number of issues and some misleading errors which was a bit challenging to fix. So, I decided to write this small post to share some of the issues that I came across and how I was was able to resolve it, so that anyone facing similar issues need not pull their hair searching for solutions.
Issue 1: Chef InSpec cannot execute without valid licenses
1. One common issue that users may encounter when running Chef InSpec with GitHub Actions, is the inability to execute tests due to missing licenses. There can be multiple reasons why you may hit this issue. But, the least expected reason is the way Chef Inspec is installed in your build container.
2. Many would use the Github Action “actionshub-chef-install“, to try and install Chef Inspec. However, this leads to the error, “Chef InSpec cannot execute without valid licenses“, despite accepting the license Chef license.
3. If you are facing a similar issue you can simply fix it by downloading and installing Chef Inspec using the cURL command rather than using any prebuilt Github actions for it.
- name: "Install Chef Inspec" run: curl https://omnitruck.chef.io/install.sh | sudo bash -s -- -P inspec |
4. In addition, do not forget to accept the Chef license. To do so, you can simply create an environment variable “CHEF_LICENSE” and set the value “accept-silent” to it.
env:
CHEF_LICENSE: accept-silent |
Issue 2: Client error, can’t connect to “ssh” backend: Your SSH Agent has no keys added, and you have not specified a password or a key file
1. Chef Inspec uses SSH for connecting to remote systems and running the tests. For this, we need the authentication key used to connect to the host system. We usually store this key in the Github Secrets. However, when we try passing this Secret directly in the SSH command like below we would hit an error:
inspec exec survey_frontend_tests --input-file ubuntu_tests/input.yaml -t ssh://ubuntu@12.345.67.890 -i ${{ secrets.SURVEY_FRONTEND_KEY }} |
The above command would give us a client error:”can't connect to 'ssh' backend: Your SSH Agent has no keys added, and you have not specified a password or a key file“.
2. This is because the SSH command cannot directly read the key from the Secrets and the key needs to be stored in a new file with the right permissions. We can do this by simply adding a step in our workflow to save the key from the Secrets to a new file and setting the required permissions for it
- name: Set the Instance key run: | echo "${{ secrets.SURVEY_FRONTEND_KEY }}" | tr -d '\r' > key.pem chmod 400 key.pem |
3. We can then use this newly created key file in our SSH command to resolve our issue.
inspec exec survey_frontend_tests --input-file survey_frontend_tests/input.yaml -t ssh://ubuntu@12.345.67.890 -i key.pem |
Issue 3: GitHub workflow failing when there are skips in your tests
1. Another interesting issue that I stumbled upon is that, the GitHub workflow fails when any of the tests in our Chef InSpec profile is skipped. You may be wondering why this happens. This is because when there are skipped tests, the exit status of your test run would be ‘101‘.
2. GitHub workflow interprets any status code other than 0 as a failure. Therefore when there are skipped tests in your test run the overall workflow status gets marked as failed.
3. We can resolve this issue by setting Chef InSpec to return only two distinct exit codes. ‘0’ on failure and ‘1’ on success. We can achieve this by passing the –no-distinct-exit flag when executing our Chef InSpec tests. This ensures that the skipped tests does not cause the test run to exit with a non-zero status unless there is an actual failure.
inspec exec survey_frontend_tests --input-file survey_frontend_tests/input.yaml -t ssh://ubuntu@12.345.67.890 -i key.pem --no-distinct-exit |
Bonus: Publishing InSpec results on Netlify
1. As part of your Chef InSpec workflow, you may want to publish the test results somewhere, let’s say Netlify. To do this this we first need to save our test results into a html file. We can simply use the –reporter flag of Chef Inspec to achieve this.
inspec exec survey_frontend_tests --input-file survey_frontend_tests/input.yaml -t ssh://ubuntu@12.345.67.890 -i key.pem --no-distinct-exit --reporter=cli html:./test_report/index.html |
2. We then need to add a new site in our Netlify account and set the NETLIFY_AUTH_TOKEN and NETLIFY_SITE_ID. We won’t be going into the details of the setup procedure here. You can refer the following blog post for more information on setting up your Netlify account
3. Once we have our Netlify account setup, We can then use a Github action ‘nwtgck/actions-netlify‘ to deploy our html file on it.
- name: Deploy DataDocs to Netlify if: always() uses: nwtgck/actions-[email protected] with: publish-dir: "./test_report" production-deploy: true env: NETLIFY_AUTH_TOKEN: ${{ secrets.NETLIFY_AUTH_TOKEN }} NETLIFY_SITE_ID: ${{ secrets.NETLIFY_SITE_ID }} timeout-minutes: 1 |
I hope you found this post helpful in resolving some of your issues. Also, keep an eye out! We have more posts on testing Infrastructure coming out soon.
Hire technical testers from Qxf2
Hiring Qxf2 brings the advantage of working with technical testers who go beyond traditional test automation practices. Our diverse skill set and expertise enable us to solve critical testing challenges and help teams iterate quickly and confidently. With our focus on early-stage products and modern technical stacks, we provide valuable insights and contribute to the success of your software development projects. Contact us!
I am a QA Engineer. I completed my engineering in computer science and joined Qxf2 as an intern QA. During my internship, I got a thorough insight into software testing methodologies and practices. I later joined Qxf2 as a full-time employee after my internship period. I love automation testing and seek to learn more with new experiences and challenges. My hobbies are reading books, listening to music, playing chess and soccer.
A good article, and I have run into many of those errors myself, checkout how our team at the MITRE SAF solved some of those common challenges as well in a few alternate ways.
https://github.com/mitre/redhat-enterprise-linux-8-stig-baseline/blob/metzger_dynamic_inputs/.github/workflows/verify-ec2.yml
Feel free to reach out or ping me on the Chef Community Slack channel #inspec