Automated Pipelines using CircleCI

Photo by Sam Loyd on Unsplash

This topic was taken as a session as part of #ThursdayDevops series, we have here at Francium Tech. You can find the information about sessions in this Github repo

Before diving into CircleCI, lets see few facts about deployments,

Etsy deploys more than 50 times per day

Amazon on average deploys every 11.7 seconds

Netflix have deployed more than 100 times per day

So this makes few things obvious. There is lot of thought process and engineering practice gone behind the deployment strategy and the whole Devops process (Automation, Monitoring).

We wont go into those details (might be for another post) but we will start something simple. Automated Pipeline for deployments. Before going in we have to see basics of Continuous Integration & Continuous Deployment.

Continuous Integration

Continuous Integration (CI) is a development practice that requires developers to integrate code into a shared repository several times a day. Each check-in is then verified by an automated build, allowing teams to detect problems early

Continuous Deployment

Continuous deployment is a strategy for software releases wherein any code commit that passes the automated testing phase is automatically released into the production environment, making changes that are visible to the software’s users.

This itself is a vast topic, you can read more about it here and here. We will see a sample Pipeline setup for this automated build,

Continuous Integration & Deployment pipeline

Few points to be noted,

• For each and every checkin into version control system, this pipeline would be triggered.
• The confidence of build would be increasing as it goes through stage by stage.
• We can have approval process for higher level environments and have automated deployment to Dev environment (Remember this is an example, your requirement might vary)

Now we will come to the crux of it. Setting up this pipeline using CircleCI.

CircleCI

CircleCI is one of the hosted CI Server which integrates with Github very well.

Setting up the project

Sign in to CircleCI using Github account and it asks permission to read your projects. If you give the permission you can go to CircleCI and get all your projects listed there. Click on Set Up Project and proceed.

Circle CI project setup

Helper utilities

CircleCI mandates us to create the configuration file at this path, PROJECT_ROOT/.circleci/config.yml. So create that file at the root of your project.

If you want to test the CircleCI configuration in local you can use these commands,

circleci config validate => Validates the yaml content of the circle ci configuration

circleci local execute — job linting => Executes the job in local

If you want to setup the circleci cli in local, please read this post.

Jobs

In the above sample pipeline diagram, each and every box is a job. It runs the actual set of steps given, inside a machine. This machine which is running the steps is called as Executors. There are multiple executors (Docker, Shell, VM). We will use Docker Executor and create a hello world example.

Hello World Job

Here we declared that

• We are going to use CircleCI version 2
• Created one job called build. If we don’t have workflows (which we will talk about next) then that default job name should be build
• We used Docker executor with busybox as the image
• List of steps to run inside the Busybox container

Workflows

But above hello world example is not sufficient to build a complete pipeline. To create multiple jobs and link them together, we will create workflows.

Hello world workflow

• We created the workflow named sample_pipeline and gave 2 jobs to it.
• These two jobs, unit_test and linting would be running in parallel and one doesn’t depend on the output of other.
• Also each job would gets its own executor. In our case, both the jobs use Docker Executor.

Pipeline Setup

Here we will create multiple jobs with dependencies between them,

Pipeline setup

Few points to note here

• Both linting & unit_test jobs would still run in parallel
• Fan In: build_artifact job would be triggered only when both unit_test & linting are successful. so both the jobs linting & unit_test fan_in to the build_artifact.
• A job can have multiple steps, as given in deploy_to_qa. These steps are always executed in sequential order and always uses the same container.

Approval Process

Only missing part here is the approval process. If we don’t have that, then the whole pipeline would be running for each and every check-in, which we don’t want to have (again depends!)

Pipeline setup with approval process

For readability, I have pasted only the workflows and left out the jobs part. Again you can check the entire configuration here. Points to be noted,

• hold_for_approval is a new job with type as approval.
• deploy_to_qa would be triggered only when hold_for_approval is done

So once everything is completed, this would be the pipeline output.

Pipeline example using CircleCI

Thanks. Hope you have learnt about the basics of pipeline and CircleCI.

Francium Tech is a technology company laser focussed on delivering top quality software of scale at extreme speeds. Numbers and Size of the data don’t scare us. If you have any requirements or want a free health check of your systems or architecture, feel free to shoot an email to contact@francium.tech, we will get in touch with you!