Choosing The Right Continuous Integration Tool

A typical software development process is based on agile framework with a need to roll out quality products with high pace. Software delivery tools are evolving too to keep pace with the challenges like “high paced changing requirements”, “evolving design”, “continuous integration” with “test driven development”. Continuous Integration (CI) and Continuous Deliveries (CD) are not only the latest buzzword but also the highly effective and practical way of developing and delivering quality products. Today, we are flooded with many CI tools and choosing the right tool is becoming difficult. Many organizations take the easy route by picking the popular tool but this strategy may be riskier since the chosen CI tools may not meet the needs in long term.

This article helps in choosing the right CI tool as per the business needs. Let us first look into the popular CI tools available in the market today. The below table gives the list of CI tools (not ranked in any order) which are popular today.

	Jenkins is an open source automation server written in Java. It helps to automate the non-human part of the software development process, with continuous integration and facilitates the technical aspects of continuous delivery. It is a server-based system that runs in servlet containers such as Apache Tomcat
	TeamCity is a Java-based build management and continuous integration server from JetBrains Inc. It is a commercial software and licensed under a proprietary license.
	Travis CI is a hosted, distributed continuous integration service used to build and test software projects hosted at GitHub. Open source projects can be tested at no charge via travis-ci.org. Private projects can be tested at travis-ci.com on a fee basis.
	GoCD is an open source CI server, which is used in software development to help teams, and organizations automate the continuous delivery of software. It supports automation of entire build-test-release process from code check-in to deployment.
	Bamboo is a CI server that can be used to automate the release management for a software application by creating a continuous delivery pipeline. It is developed as Atlassian Inc. with propriety license for the private projects but free license for open source projects.
	GitLab is a web-based Git-repository manager with wiki, issue tracking and CI/CD pipeline features, using an open-source license, developed by GitLab Inc. It can be used freely for limited used for basic features but license fees is required for advance features.
	CircleCI's continuous integration and delivery platform makes it easy for teams of all sizes to rapidly build and release quality software at scale in the cloud or behind the firewall. Excellent support for Linux based software development. Supports all open source version control systems.
	Codeship is a fast and secure hosted Continuous Integration service that scales with business needs. It supports GitHub, Bitbucket, and Gitlab projects.
	Wercker empowers organizations and their development teams to achieve continuous integration and continuous delivery (CI/CD) goals with micro-services and Docker. This is achieved via Wercker’s container-centric and cloud-native automation platform comprised of their local command line interface (CLI), online SaaS platform and API. Wercker is hosted on Oracle Public Cloud for creating containerized pipelines.
Jenkins X	Jenkins X has the concepts of Apps and Environments built in. Jenkins X does not aim to replace Jenkins since Jenkins is embedded as a pipeline engine as part of the installation. It is a one of the fast emerging CI/CD solution for modern cloud applications on Docker and Kubernetes.
Shippable	Shippable is lightweight CI, CD, and DevOps automation platform with Docker support that simplifies provisioning, building, testing, and deploying any application anywhere. It comes with a propriety commercial license with support for Github and BitBucket repositories.
	NeverCode is a cloud-based CI and CD server that automates the process of building, testing and distributing mobile applications. Claims to help developers develop apps 20% faster with the leading CI/CD solutions.

Essential Technical Features

A CI tool must have features, which caters to both on-premises and cloud development. Few of such technical features are as follows:

Security Enablement	Role based access control, SSO based authentication, CSRF Protection, Slave/Node/External Runners to Master Access Control, protection from malicious builds and users
Container Support	Support to build and pull Docker images from various Docker repositories (both private and public), Support for Kubernetes cloud, running builds on Kubernetes nodes, support of persistent storage of logs and workspace, deployment of built Docker images to Kubernetes managed cloud
Version Control Software Support	Support for various version control system and repositories like Subversion, Perforce, Git, Rational Clear Case, Razor, SourceAnywhere etc. hosted on-premises or on cloud.
Easy Complex Pipelines	Ease to create complex pipelines. There could be requirements which could result in a complex release steps and a CI tool should meet these requirments. Such requirements might mandate pipeline to have multiple forks and joins based on conditions and trigger remote pipelines and workflows.
Good User Interface	User interface should be web browser based to check the build consoles, build configuration, pipeline code YAML file, build time history, build marking with various badges, support for various views like pipeline view, job view, workflow view etc, Instant report views for various builds, builds comparison, ease of global configuration, ease of doing third party tools configuration etc.
Notifications Support	Support for notifications on various integration stages via Email, Hipchat, Slack, Stride, Flowdock, Campfire etc.
Support for Build Automation Tool	Different projects have different type of build tools requirement. A CI tool must be flexible enough to support all types of build tools like Make, Shell Scripts, Ant, Maven, Gradle, Rake, Grunt, Gulp etc.
Artifacts Repository Support	Feature to store, retrieve the build information, build/test’s artifacts pull/push from/to Apache Nexus , JFrog Artifactory or any other artifacts repositories via plugins, steps or REST calls.
Multiple Programing Language Support	Support for various programming languages like Node.js, Java, Ruby, Python, Golang, Groovy, Shell Scripts etc.
Integrations with Public Clouds	It should be able to do pipeline deployments to various Oracle Cloud – Classic, Oracle Cloud Infrastructure, Google Compute engine and AWS with ease.
Monitoring Support	Monitoring of various build environment parameters like slave’s memory, free disk space, builds, number of executor threads, number of nodes , time sync with master etc.
Integration with Third Party Software	Automated filing and updating of bugs to JIRA, incident filing in Pager Duty and BugZilla. Integration of static analysis tools, legal compliance tools for open source and for the in-house built artifacts.
Library of Re-usable Code	Public store of various plugins, usable build steps, which could  be open sourced or commercially available.

Business Requirements

Apart from the technical features, it is also essential to compare every CI tool with respect to the business specific requirements. Below are few such points that could be useful for such evaluations.

Open Source vs. Commercial

The main driver of decision here is the budget. Choose one of the popular CI tool if budget is available for license fees and there is a need for online/offline support for pipeline maintenance. Teamcity, CircleCI, Bamboo, GitLab-EE, Wercker and Codeship are few of the examples of popular CI tool in this category. Some of these tools charge fees based on the number of users, remote agents/nodes/runners and number of features while some have fixed license fees. Utmost care should be taken to not be stuck with a CI tool that does not have a good plan to adopt to emerging development technologies.

Alternatively, it makes sense to go for CI tools, which are open source like Jenkins. These are mostly community driven projects with lot of plugins and support available online via tutorials, blogs, chats and forums. There are organizations which provide commercial support for open source CI tool, for example, CloudBees Inc. provide commercial support for the Jenkins.

On Premises vs. Cloud Hosted

Cloud software architecture is quite different from an on-premises traditional software development. Cloud based solutions are largely based on micro services architecture with software pieces strewn across the Internet. They are implemented using container tools like Docker and deployed to Kubernetes via offerings from Google, Amazon or Oracle. Tools like Gitlab and Wercker are perfect for Docker based micro services with GKE or custom Kubernetes support where as tools like Jenkins can be enhanced via the plugins to cater for these use cases.

CI servers like Jenkins requires an on-premises server hosting with slaves/nodes (virtual machines) attached to it. This requires efforts to install, monitor and maintain the system.  However, a cloud-hosted solution is mostly ready to use with minimal configuration efforts, offers infrastructure reliability and can be scaled up/down based on the job load. Operation costs of an on-premises hosting could be less expensive than the cloud hosted solution. Other deciding criteria could be legal and statutory requirements where the artifacts or data should not be posted to cloud but should be accessible to the compliance authorities locally. For smaller setups or startups, the cloud based hosted CI solution is better where basic essential features are available for small set of users with minimal or no fees.

Choose Multiple CI Tools

Can one size really fit all? Not really. A popular CI tool may meet 80% of the automation requirements but not fit the other 20%. Jenkins, for example, can meet almost all the requirements but it requires manual effort to monitor the Jenkins server, its slave, understand inter dependencies amongst plugins and upgrade various plugins from time to time. There are CI tools like Wrecker which works perfectly if a product is cloud based but may require support for an on-premises software product.

An organization could have multiple projects of different kinds, which could range from a monolithic big software system to a micro service based architecture and one CI tool may not fit all the scenarios. So, it is better to choose multiple CI tools for various needs instead of fitting all into one CI tool. Such a strategy is useful for the business continuity too where projects will not be impacted if a support from a CI tool is not good or the CI tool is discontinued.

Migration from a CI server to Other

A CI pipeline could be defined “as a code” that directs the CI server to use a specific build environment, build steps, easy integration with third party applications etc. Most of the pipeline as a code uses YAML format, which could be compatible to other CI tools too. However, custom inbuilt YAML step might not be portable to other CI tools, so, care should be taken to implement the build and deployment steps as generic as possible so that if the need arise, the entire pipeline could be shifted to another CI server.

Conclusion

Yes, there are a lot of CI/CD tools available today and it is difficult to choose a best amongst many considering the fact that many tools do not provide all the required features. Therefore, it is suggested to choose a CI solution with the help of the pointers given in this article with a condition that it should not only meet the current requirements but should also evolve along with your product’s future roadmap.