CircleCI nightly automation

A Qxf2 client is using CircleCI for their continuous integration and deployment needs. We decided to run a small subset of critical tests in our automation suite against every new commit and deploy. We also run our entire automation suite, which takes longer, once every night. This post outlines how we setup CircleCI to achieve this setup.

CirleCI event and time triggered tests
Figure 1. How we wanted to trigger tests in our test suite

Why this post?

CircleCI seems to be growing in popularity as an alternative to Jenkins. It took us a while to explore and get used to CircleCI. We thought we would share some of our learnings.


Setup

For this tutorial we will have one repository for all our application code and a separate repository for our tests. We are doing this to make the example more realistic. Please refer to our earlier post if you want to know how to integrate your GitHub repository with CircleCI.

a) Application repository
To keep things simple, let us add a single html page in our GitHub application repository. I added an html page which we had created earlier as part of Selenium training. You can get the source from this link. I will name the repository as “selenium_tutorial”. Once you check in this repository you need to give CircleCI permission to access this repository.

b) Test repository
Let us pretend that our automated test suite has three tests: Navigate_Qxf2_Tutorial_Test.py, Test1.py and Test2.py. We do not want to run Test1.py and Test2.py after every build. We only want to run Test1.py and Test2.py once every night.

Let us add a simple Selenium script using Python to navigate to the Qxf2 Selenium Tutorial page and assert the page title. Once you have written the test, check-in your changes to a separate “selenium_tutorial_test” repository.

"""
Selenium Test to login to Qxf2 Tutorial Page and assert the title
"""
from selenium import webdriver
 
# Create an instance of Firefox WebDriver
driver = webdriver.Firefox()
# The driver.get method will navigate to a page given by the URL
driver.get("http://localhost/selenium-tutorial-main.html")
# Assert the Page Title
assert "Qxf2 Services: Selenium training main" in driver.title
# Close the browser window
driver.close()

Event based automation runs

To trigger our Navigate_Qxf2_Tutorial_Test.py after every build of our application repository, we simply need to add the following lines to the circle.yml file of the application test repository.

## Customize dependencies
dependencies:
  pre:
    - sudo pip install -U selenium
    - sudo apt-get update
    - sudo apt-get install nginx
     # Copy the html page to nginx default public www location
    - sudo cp ./selenium-tutorial-main.html /var/www/html/selenium-tutorial-main.html
 
## Running Tests
test:
  # The key call to clone the test repository before we run the test
  pre:
    - git clone https://github.com/Qxf2Circle/selenium_tutorial_test.git
 
  override:
    - python selenium_tutorial_test/Navigate_Qxf2_Tutorial_Test.py

We ensure that Navigate_Qxf2_Tutorial_Test.py test runs after every build by specifying it in the override section


CircleCI nightly automation runs

We want to run Test1.py and Test2.py once every night. To make sure that they do not run after every build, we need to perform the following steps:
1. Setup an API token
2. Write a shell script that triggers a nightly build by calling a CircleCI API
3. Modify circle.yml file
4. Setup a cronjob to kick off the .sh script regularly

1. Setup an API token
You can create an API token in your ‘Account Settings’ page. To create an API token, simply give it a name and click on the ‘Create new token’ button. For now, copy your API token and keep it handy.

CircleCI_API_Token

2. Write a shell script that triggers a nightly build by calling a CircleCI API
CircleCI provides an API which lets you kick-off builds. The API (simply a URL that you POST to) needs to know your project,branch and circle token. For this tutorial we will create a nightly_build.sh file which takes these three parameters as command line arguments.

#!/bin/bash
#Src: https://circleci.com/docs/nightly-builds
 
_project=$1
_branch=$2
_circle_token=$3
 
trigger_build_url=https://circleci.com/api/v1/project/${_project}/tree/${_branch}?circle-token=${_circle_token}
 
post_data=$(cat <<EOF
{
  "build_parameters": {
    "NIGHTLY_BUILD": "true"
  }
}
EOF)
 
curl \
--header "Accept: application/json" \
--header "Content-Type: application/json" \
--request POST ${trigger_build_url}

Your project name is typically /. For example, Qxf2Circle/selenium_tutorial. Your CircleCI token is what you created in Step 1. To trigger the nightly run, you would run the command:

nightly_build.sh <your_project_name> master <your_circleci_token>

3. Modify your circle.yml file
To your existing circle.yml file, add logic to trigger Test1.py and Test2.py only if the POST has the NIGHTLY_BUILD parameter to True.

## Customize dependencies
dependencies:
  pre:
    - sudo pip install -U selenium
    - sudo apt-get update
    - sudo apt-get install nginx
     # Copy the html page to nginx default public www location
    - sudo cp ./selenium-tutorial-main.html /var/www/html/selenium-tutorial-main.html
 
## Running Tests
test:
  # The key call to clone the test repository before we run the test
  pre:
    - git clone https://github.com/Qxf2Circle/selenium_tutorial_test.git
 
  override:
    - python selenium_tutorial_test/Navigate_Qxf2_Tutorial_Test.py
  post:
    - >
      if [ -n "${NIGHTLY_BUILD}" ]; then
         python ./selenium_tutorial_test/Test1.py
         python ./selenium_tutorial_test/Test2.py 
      fi

4. Setup a cronjob to kick off the .sh script regularly
To kick of the script we need to invoke scheduled job. To run the new job every night we will create a cron job by running crontab -e and adding:

30 0 * * * /bin/bash /home/avinash/Downloads/nightly_build.sh Qxf2Circle/selenium_tutorial master $CIRCLE_TOKEN

This runs the nightly_build.sh script at 30 minutes past midnight with arguments

_project=Qxf2Circle/selenium_tutorial
_branch=master
_circle_token=Update your circleci token here

If you wanted to trigger a run of the entire suite sometime during the day, simply call:

nightly_build.sh Qxf2Circle/selenium_tutorial master $CIRCLE_TOKEN

And with that you know how to use CircleCI to run a small subset of your tests after every build, while running your entire automation suite every night. Let us know if you have any questions in the comments below!

If you are a startup finding it hard to hire technical QA engineers, learn more about Qxf2 Services.


Subscribe to our weekly Newsletter


View a sample



10 thoughts on “CircleCI nightly automation

  1. pretty useful. I just moved from jenkins to circle, something we can do it jenkins but I cannot find it in circle yet.

    But your blog is more helpful than the circle doc 😛

    thanks very much!

  2. You have written to run a command to schedule a job. But on which terminal we have to fire the command. Means “cronjob -e ” needs to be run for scheduling but where we have to write it.

  3. Hi,
    I have setup a cron job to run my Appium UI tests on circle CI and the tests end up running on the dashboard screen of simulator which makes the test fail. It ran fine locally.

    Below is the Circle.yml test sec:
    test:
    override:
    – >
    if [ -n “${RUN_APPIUM_TESTS}” ]; then
    # Install java jdk8
    brew update
    brew cask install java
    # Install maven
    brew install maven
    npm install -g [email protected]
    xcrun instruments -w “iPhone 8 Plus (11.0.1) [” || true
    xcodebuild CODE_SIGNING_REQUIRED=NO CODE_SIGN_IDENTITY= PROVISIONING_PROFILE= -scheme “XXXX” -workspace ‘XXXXX’ -sdk ‘iphonesimulator’ -destination ‘platform=iOS Simulator,name=iPhone 8 Plus,OS=11.0.1’ | xcpretty
    #cd to Code directory & run tests!
    pushd “appium-selenium/MumboiOSTests”
    appium &
    mvn clean test;
    popd
    fi

    build setup in a linux instance is as below:

    _project=$1
    _branch=$2
    _circle_token=$3

    trigger_build_url=https://circleci.com/api/v1.1/project/github/${_project}/tree/${_branch}?circle-token=${_circle_token}

    post_data=$(cat <

    @BeforeClass
    public void setup() throws MalformedURLException {

    File app = new File(“build”);
    File source = new File(app, “Mumbo.app”);
    DesiredCapabilities ios = new DesiredCapabilities();
    ios.setCapability(MobileCapabilityType.DEVICE_NAME, “iPhone 8 Plus”);
    ios.setCapability(MobileCapabilityType.PLATFORM_NAME, “IOS”);
    ios.setCapability(MobileCapabilityType.PLATFORM_VERSION, “11.0”);
    ios.setCapability(MobileCapabilityType.AUTOMATION_NAME, AutomationName.IOS_XCUI_TEST);
    ios.setCapability(MobileCapabilityType.APP, source.getAbsolutePath());
    ios.setCapability(“useNewWDA”, true);
    driver = new IOSDriver(new URL(“http://127.0.0.1:4723/wd/hub”), ios);
    }

    I wonder if I’m missing something on the configuration side. Is there a platform where I can attach the appium and ios logs. Kindly take a look

    1. Hi Allie,
      Thanks for providing circle.yml file details and other configuration details along with your test build explanation. It would be more helpful if you could provide error message also.
      As per my observation, there may be a problem with running Appium, as a command “appium &” won’t work properly on CircleCI. To run the Appium server in the background, you need to set background flag and provide some time to the server to start up. Use below lines for it.
      – appium:
      background: true
      -sleep 5
      You can attach the Appium & iOS logs, test reports, screenshots on CircleCI as artifacts.
      Refer to the following doc for circle.yml configuration details for iOS and getting logs on CircleCI.
      https://circleci.com/docs/1.0/ios-builds-on-os-x/

  4. Thanks for your reply. I was able to fix the issue, the xcodebuild was having an issue. Also, is there a way we could email the test artifacts/outputs from Circle CI for just the builds triggered from the cron job.

    1. Hi Allie,
      Looks like, there is no straightforward way to email the test artifacts/outputs from Circle CI for just(only) the builds triggered from the cron job.
      You need to enhance your test framework to send an email along with the test artifacts/output.
      You can refer our pythonic utility file which we use to send test report emails at
      https://github.com/qxf2/qxf2-page-object-model/blob/master/utils/email_pytest_report.py
      Based on your circle.yml details, I am sure, your test was written in Java. The following link may help you to build email utility http://learn-automation.com/send-report-through-email-in-selenium-webdriver/
      Once, you are done with email test report utility. You need to add a command to run tests and send the email report in circle.yml file below cron job token/flag.

Leave a Reply

Your email address will not be published. Required fields are marked *