A Qxf2 client is using CircleCI for their continuous integration and deployment needs. We decided to run a small subset of critical tests in our automation suite against every new commit and deploy. We also run our entire automation suite, which takes longer, once every night. This post outlines how we setup CircleCI to achieve this setup.
Why this post?
CircleCI seems to be growing in popularity as an alternative to Jenkins. It took us a while to explore and get used to CircleCI. We thought we would share some of our learnings.
Setup
For this tutorial we will have one repository for all our application code and a separate repository for our tests. We are doing this to make the example more realistic. Please refer to our earlier post if you want to know how to integrate your GitHub repository with CircleCI.
a) Application repository
To keep things simple, let us add a single html page in our GitHub application repository. I added an html page which we had created earlier as part of Selenium training. You can get the source from this link. I will name the repository as “selenium_tutorial”. Once you check in this repository you need to give CircleCI permission to access this repository.
b) Test repository
Let us pretend that our automated test suite has three tests: Navigate_Qxf2_Tutorial_Test.py, Test1.py and Test2.py. We do not want to run Test1.py and Test2.py after every build. We only want to run Test1.py and Test2.py once every night.
Let us add a simple Selenium script using Python to navigate to the Qxf2 Selenium Tutorial page and assert the page title. Once you have written the test, check-in your changes to a separate “selenium_tutorial_test” repository.
""" Selenium Test to login to Qxf2 Tutorial Page and assert the title """ from selenium import webdriver # Create an instance of Firefox WebDriver driver = webdriver.Firefox() # The driver.get method will navigate to a page given by the URL driver.get("http://localhost/selenium-tutorial-main.html") # Assert the Page Title assert "Qxf2 Services: Selenium training main" in driver.title # Close the browser window driver.close() |
Event based automation runs
To trigger our Navigate_Qxf2_Tutorial_Test.py after every build of our application repository, we simply need to add the following lines to the circle.yml file of the application test repository.
## Customize dependencies
dependencies:
pre:
- sudo pip install -U selenium
- sudo apt-get update
- sudo apt-get install nginx
# Copy the html page to nginx default public www location
- sudo cp ./selenium-tutorial-main.html /var/www/html/selenium-tutorial-main.html
## Running Tests
test:
# The key call to clone the test repository before we run the test
pre:
- git clone https://github.com/Qxf2Circle/selenium_tutorial_test.git
override:
- python selenium_tutorial_test/Navigate_Qxf2_Tutorial_Test.py |
We ensure that Navigate_Qxf2_Tutorial_Test.py test runs after every build by specifying it in the override section
CircleCI nightly automation runs
We want to run Test1.py and Test2.py once every night. To make sure that they do not run after every build, we need to perform the following steps:
1. Setup an API token
2. Write a shell script that triggers a nightly build by calling a CircleCI API
3. Modify circle.yml file
4. Setup a cronjob to kick off the .sh script regularly
1. Setup an API token
You can create an API token in your ‘Account Settings’ page. To create an API token, simply give it a name and click on the ‘Create new token’ button. For now, copy your API token and keep it handy.
2. Write a shell script that triggers a nightly build by calling a CircleCI API
CircleCI provides an API which lets you kick-off builds. The API (simply a URL that you POST to) needs to know your project,branch and circle token. For this tutorial we will create a nightly_build.sh file which takes these three parameters as command line arguments.
#!/bin/bash #Src: https://circleci.com/docs/nightly-builds _project=$1 _branch=$2 _circle_token=$3 trigger_build_url=https://circleci.com/api/v1/project/${_project}/tree/${_branch}?circle-token=${_circle_token} post_data=$(cat <<EOF { "build_parameters": { "NIGHTLY_BUILD": "true" } } EOF) curl \ --header "Accept: application/json" \ --header "Content-Type: application/json" \ --request POST ${trigger_build_url} |
Your project name is typically
nightly_build.sh <your_project_name> master <your_circleci_token> |
3. Modify your circle.yml file
To your existing circle.yml file, add logic to trigger Test1.py and Test2.py only if the POST has the NIGHTLY_BUILD parameter to True.
## Customize dependencies
dependencies:
pre:
- sudo pip install -U selenium
- sudo apt-get update
- sudo apt-get install nginx
# Copy the html page to nginx default public www location
- sudo cp ./selenium-tutorial-main.html /var/www/html/selenium-tutorial-main.html
## Running Tests
test:
# The key call to clone the test repository before we run the test
pre:
- git clone https://github.com/Qxf2Circle/selenium_tutorial_test.git
override:
- python selenium_tutorial_test/Navigate_Qxf2_Tutorial_Test.py
post:
- >
if [ -n "${NIGHTLY_BUILD}" ]; then
python ./selenium_tutorial_test/Test1.py
python ./selenium_tutorial_test/Test2.py
fi |
4. Setup a cronjob to kick off the .sh script regularly
To kick of the script we need to invoke scheduled job. To run the new job every night we will create a cron job by running crontab -e and adding:
30 0 * * * /bin/bash /home/avinash/Downloads/nightly_build.sh Qxf2Circle/selenium_tutorial master $CIRCLE_TOKEN |
This runs the nightly_build.sh script at 30 minutes past midnight with arguments
_project=Qxf2Circle/selenium_tutorial
_branch=master
_circle_token=Update your circleci token here
If you wanted to trigger a run of the entire suite sometime during the day, simply call:
nightly_build.sh Qxf2Circle/selenium_tutorial master $CIRCLE_TOKEN |
And with that you know how to use CircleCI to run a small subset of your tests after every build, while running your entire automation suite every night. Let us know if you have any questions in the comments below!
If you are a startup finding it hard to hire technical QA engineers, learn more about Qxf2 Services.
I am a dedicated quality assurance professional with a true passion for ensuring product quality and driving efficient testing processes. Throughout my career, I have gained extensive expertise in various testing domains, showcasing my versatility in testing diverse applications such as CRM, Web, Mobile, Database, and Machine Learning-based applications. What sets me apart is my ability to develop robust test scripts, ensure comprehensive test coverage, and efficiently report defects. With experience in managing teams and leading testing-related activities, I foster collaboration and drive efficiency within projects. Proficient in tools like Selenium, Appium, Mechanize, Requests, Postman, Runscope, Gatling, Locust, Jenkins, CircleCI, Docker, and Grafana, I stay up-to-date with the latest advancements in the field to deliver exceptional software products. Outside of work, I find joy and inspiration in sports, maintaining a balanced lifestyle.
pretty useful. I just moved from jenkins to circle, something we can do it jenkins but I cannot find it in circle yet.
But your blog is more helpful than the circle doc 😛
thanks very much!
You have written to run a command to schedule a job. But on which terminal we have to fire the command. Means “cronjob -e ” needs to be run for scheduling but where we have to write it.
Hi Rohit,
CRON is an utility in Linux with which you can start your jobs automatically at a desired time and schedule them to get executed periodically. Please refer to below link for more details
http://www.yourownlinux.com/2014/04/schedule-your-jobs-in-linux-with-cron-examples-and-tutorial.html
https://discuss.circleci.com/t/how-to-run-cron-job-on-circle-ci/2652
CircleCI Employee: “`You cannot run CRON on circleci, you would need to set up a CRON job on one of your own servers (or any computer that is always powered on) in order to get nightly builds to work.“`
Unfortunately you have to think of third-party scheduling service.
Denis, If you have aws account you can use the following:
https://github.com/godfreyhobbs/circleci-cron-serverless
It does not require a server to run cron.
Hi,
I have setup a cron job to run my Appium UI tests on circle CI and the tests end up running on the dashboard screen of simulator which makes the test fail. It ran fine locally.
Below is the Circle.yml test sec:
test:
override:
– >
if [ -n “${RUN_APPIUM_TESTS}” ]; then
# Install java jdk8
brew update
brew cask install java
# Install maven
brew install maven
npm install -g [email protected]
xcrun instruments -w “iPhone 8 Plus (11.0.1) [” || true
xcodebuild CODE_SIGNING_REQUIRED=NO CODE_SIGN_IDENTITY= PROVISIONING_PROFILE= -scheme “XXXX” -workspace ‘XXXXX’ -sdk ‘iphonesimulator’ -destination ‘platform=iOS Simulator,name=iPhone 8 Plus,OS=11.0.1’ | xcpretty
#cd to Code directory & run tests!
pushd “appium-selenium/MumboiOSTests”
appium &
mvn clean test;
popd
fi
build setup in a linux instance is as below:
_project=$1
_branch=$2
_circle_token=$3
trigger_build_url=https://circleci.com/api/v1.1/project/github/${_project}/tree/${_branch}?circle-token=${_circle_token}
post_data=$(cat <
@BeforeClass
public void setup() throws MalformedURLException {
File app = new File(“build”);
File source = new File(app, “Mumbo.app”);
DesiredCapabilities ios = new DesiredCapabilities();
ios.setCapability(MobileCapabilityType.DEVICE_NAME, “iPhone 8 Plus”);
ios.setCapability(MobileCapabilityType.PLATFORM_NAME, “IOS”);
ios.setCapability(MobileCapabilityType.PLATFORM_VERSION, “11.0”);
ios.setCapability(MobileCapabilityType.AUTOMATION_NAME, AutomationName.IOS_XCUI_TEST);
ios.setCapability(MobileCapabilityType.APP, source.getAbsolutePath());
ios.setCapability(“useNewWDA”, true);
driver = new IOSDriver(new URL(“http://127.0.0.1:4723/wd/hub”), ios);
}
I wonder if I’m missing something on the configuration side. Is there a platform where I can attach the appium and ios logs. Kindly take a look
Hi Allie,
Thanks for providing circle.yml file details and other configuration details along with your test build explanation. It would be more helpful if you could provide error message also.
As per my observation, there may be a problem with running Appium, as a command “appium &” won’t work properly on CircleCI. To run the Appium server in the background, you need to set background flag and provide some time to the server to start up. Use below lines for it.
– appium:
background: true
-sleep 5
You can attach the Appium & iOS logs, test reports, screenshots on CircleCI as artifacts.
Refer to the following doc for circle.yml configuration details for iOS and getting logs on CircleCI.
https://circleci.com/docs/1.0/ios-builds-on-os-x/
Thanks for your reply. I was able to fix the issue, the xcodebuild was having an issue. Also, is there a way we could email the test artifacts/outputs from Circle CI for just the builds triggered from the cron job.
Hi Allie,
Looks like, there is no straightforward way to email the test artifacts/outputs from Circle CI for just(only) the builds triggered from the cron job.
You need to enhance your test framework to send an email along with the test artifacts/output.
You can refer our pythonic utility file which we use to send test report emails at
https://github.com/qxf2/qxf2-page-object-model/blob/master/utils/email_pytest_report.py
Based on your circle.yml details, I am sure, your test was written in Java. The following link may help you to build email utility http://learn-automation.com/send-report-through-email-in-selenium-webdriver/
Once, you are done with email test report utility. You need to add a command to run tests and send the email report in circle.yml file below cron job token/flag.
Thank you! Appreciate your help. I’ll try it out 🙂