Jenkins vs Travis CI vs Circle CI vs TeamCity vs Codeship vs GitLab CI vs Bamboo

 ● 24th Oct 2017

13 min read

Why should you use a tool for your CI/CD workflow, and which one is the right tool for you?


More and more engineering teams are adopting agile development, pushing for shorter and faster release cycles. Code base growth and the frequency of creating new production build led to the rise of Continuous Integration and Continuous Deployment/Delivery tools.

If you’re also thinking of making the switch to a fast release frequency, or if you’re not certain what other tools have to offer – we’ve got you covered. In the following post we’ll get familiar with some of the most popular CI/CD tools out there, and get to know each one a little bit better. Ship ahoy!

Psst! Faster releases are great, but not if they negatively affect your application’s quality and reliability. Set up automated quality gates with OverOps to stop bad code from reaching production.

Table of contents

  1. Jenkins
  2. Travis CI
  3. Circle CI
  4. TeamCity
  5. Codeship
  6. GitLab CI
  7. Bamboo

What’s CI/CD and what is it good for?

Before taking a deep dive into CI/CD automation tools, we first need to understand the concept at large. As we mentioned, Continuous Integration and Continuous Deployment usually go hand in hand with an agile development environment, where teams want to deploy different pieces of code to production as soon as they’re finished with them.

The use of CI/CD tools automates the process of building, testing and deploying new code. Each team member can get immediate feedback about the production readiness of their code, even if they just changed a single line or character. That way, each team member can push their code into production, while the process of building, testing and deploying is done automatically so they can move on to work on the next part of the application.

Adding automation to your workflow doesn’t end with deploying code into production. You have to keep track of new errors before they significantly impact your users. For most teams, debugging in production is a manual and tedious task that requires their full attention for hours and days of sifting through logs. However, now it’s possible to automate root cause analysis at scale, knowing where, when and most importantly, why errors are happening.

If you’re a Java, Scala or a .NET developer, we have a special treat for you, check it out.

Now that we know why it’s important to implement automation in our workflow with a CI model, it’s time to see which tool is the right one for us.

1. Jenkins

Jenkins is one of the more known and common names in the CI market. It started as a side project by one of Sun’s engineers, and expanded into one of the biggest open source CI tools that helps engineering teams automate their deployments. Full disclosure: we at OverOps also use Jenkins, along with a homegrown CLI tool.

What does it do?

Just like a CI tool promises, with Jenkins you can automate your build, test and deploy tasks. The tool supports Windows, Mac OSX and various Unix systems, and can be installed using native system packages, as well as Docker or installed as a standalone on any machine with a Java Runtime Environment (JRE) installed.

On the practical side, Jenkins gives any member of the team the ability to push their code to the build, and get immediate feedback on whether it’s ready for production or not. In most cases, this will require some tinkering and tailoring of Jenkins according to your team’s custom requirements.

The place where Jenkins shines is with its rich plugin ecosystem. It offers an extended version with over 1,000 plugins, which allows integration with almost every tool and service that’s available on the market. Being an open source tool also gives you the option to custom fit it for a home-grown solution, just like we did. However, the need to spend time and some effort to make sure it is suitable for you might be a downside for some teams.

One more thing: We’ve said it once and we’ll say it again: open source + plugins = community. Any configuration, workflow, need or desire you can think of, you’ll have an option to create it with the help of Jenkins and its plugins. Also, great name for a band.

Bottom line: If you’re looking for a cheap (free!) CI solution, are willing to put in the work to customize your environment and need support from a community of users, Jenkins is the choice for you.

jenkins job list
Standard Jenkins jobs list

2. Travis CI

Travis CI is one of the more common names in the CI/CD ecosystem, created for open source projects and then expanded to closed source projects over the years. It’s focused on the CI level, improving the performance of the build process with automated testing and an alert system.

What does it do?

Travis-CI focuses on allowing users to quickly test their code as it’s deployed. It supports large and small code changes, and is designed to identify changes in building and testing. When a change is detected, Travis CI can provide feedback whether the change was successful or not.

Developers can use Travis CI to watch the tests as they run, run a number of tests in parallel, and integrate the tool with Slack, HipChat, Email and so on to get notified of issues or unsuccessful builds.

Travis CI supports container builds, and supports Linux Ubuntu and OSX. You can use it across different programming languages, such as Java, C#, Clojure, GO, Haskell, Swift, Perl and much more. It has a limited list of third party integrations, but since the focus is on CI rather than CD, it might not be an issue for your use case.

One more thing: To make sure you always have a backup of your recent build, Travis CI clones your GitHub repository into a new virtual environment whenever you run a new build.

Bottom line: If your code is open source and you’re more concerned about the continuous integration of your builds, Travis CI is worth a try.

travis ci dashboard
The Travis CI dashboard

3. Circle CI

Circle CI is a cloud based tool that automates the integration and deployment process. It also focuses on testing every change to the code before it’s deployed, using a number of methods such as unit tests, integrations tests and functional tests. The tool supports containers, OSX, Linux and can run within a private cloud or your own data center.

What does it do?

Circle CI integrates with your current version control system, such as GitHub, Bitbucket and others, and runs a number of steps whenever a change is detected. These changes could be commits, opening PRs or any other change to the code.

Every code change creates a build and runs the tests in a clean container or in a VM, according to your initial configurations and preferences. Each build consists of a number of steps, including dependencies, testing and deployment. If the build passes the tests, it can be deployed through AWS CodeDeploy, Google Container Engine, Heroku, SSH or any other method of your choice.

The success or failure status of the builds and tests in question is sent via Slack, HipChat, IRC or a number of other integrations, so the team can stay updated. It’s important to note that Circle CI requires some tweaks and changes for a number of languages, so it’s best to go over the documentation for your language of choice.

One more thing: Circle CI can auto-cancel redundant builds on GitHub. If a newer build is triggered on the same branch, the tool identifies it and cancels the older builds that are running or queued, even if the build in not finished.

Bottom line: If you’re looking for a GitHub friendly tool with a wide community behind it, that can also run within a private cloud or your inside own data center, Circle CI is worth checking out.

circle ci dashboard
The Circle CI dashboard

4. TeamCity

TeamCity is a CI/CD server, made by JetBrains. It offers continuous integration “out of the box”, as well as allowing users to best fit the tool to their own needs. It offers support for a number of languages (Java, .NET, Ruby and others), and has JetBrains to back the tool up support and documentation wise.

What does it do?

As a CI/CD tool, TeamCity aims to improve release cycles. With it you can review on-the-fly of test results, see code coverage and find duplicates, as well as customize statistics on build duration, success rate, code quality and other custom metrics.

Once TeamCity detects a change in your version control system, it adds a build to the queue. The server finds an idle compatible build agent and assigns the queued build to this agent, which executes the build steps.

While this process is running TeamCity server logs the different log messages, test reports, and other changes that are being made. The changes are saved and uploaded in real time, so users can know what’s happening with the build as they make changes to it. The tool also offers an option to run parallel builds simultaneously on different platforms and environments.

One more thing: TeamCity comes with an option of gated commits, which can prevent developers from breaking sources in a version control system. This is done by running the build remotely for local changes, before committing it.

Bottom line: TeamCity has been gaining popularity over the past few years, offering a decent alternative to other CI tools in the market. And if you’re interested in viewing your builds and tests as they happen, or want a free and powerful CI solution, there’s no doubt that TeamCity is worth checking out.

teamcity dashboard
The TeamCity dashboard

5. Codeship

Codeship has a different view on CI/CD, and offers a Hosted Continuous Integration as a Service. This tool was originally built to give a Continuous Integration platform for Rails developers; hosting their code on GitHub and deploying to Heroku. Due to its popularity and demand, the company expanded over the years to support other technologies as well.

What does it do?

Codeship has 2 different variations of the product, each with its pros and cons. Codeship Basic allows setting up a CI/CD process by connecting a repository through a web UI and turnkey deployments. It offers support for a pre-configured CI environment, and allows multiple different builds to run on the same build VM.

Codeship Pro uses Docker to define a CI/CD environment, through it you can run your build pipeline. It comes with complete control over the build environment, letting you define what to run within it. The Pro version also allows pre branch caching, to set which image and which part of the workflow gets cached, along with parallel deployments.

As a whole, Codeship supports a number of languages, such as Java, Go, Node.js, Python, Ruby and others. When it comes to deployment, the Basic version supports AWS, Heroku, Azure and Kubernetes while the Pro also supports AWS ElasticBeanstalk, Google App Engine and DigitalOcean.

One more thing: Codeship has a public collection of utilities, scripts and Docker images to use with the tool, and the company even points out that some of these can be used with other similar tools. This collection includes deployment scripts for external services that are customizable, scripts to install specific versions of software that are not included by default on the build VMs and more.

Bottom line: Offering 2 different tools under one domain might seem a bit odd, but it gives Codeship the option to focus on various elements that will better suit different kinds of customers. Since both the Basic and Pro are available for free, it’s an interesting choice for you CI needs.

codeship dashboard
The Codeship dashboard

6. GitLab CI

Soon after GitLab was launched, the team launched GitLab CI, the continuous integration service. Along with testing and building projects, the tool can also deploy the builds to your infrastructure, giving you an option to track your different deployments by knowing where each piece of code went.

What does it do?

GitLab CI is offered for free for as a part of GitLab, and can be set up rather quickly. To start using GitLab CI, you first need to add a .gitlab-ci.yml file to the root directory of your repository, as well as configure your GitLab project to use a Runner. Afterwards, every commit or push will trigger the CI pipeline that has three stages: build, test and deploy.

Each built can be divided into multiple jobs, and can run in parallel on multiple machines. The tool gives immediate feedback on the success or failure of the build, letting users know whether something went wrong or if something broke in the process.

One more thing: GitLab (and GitLab CI) is an open source project. In other words, you have access and the ability to modify the GitLab Community Edition, and the Enterprise Edition source code.

Bottom line: If you’re using GitLab, it’s almost a no-brainer to try the GitLab CI solution as part of the pack.

gitlab ci dashboard
The GitLab CI dashboard

7. Bamboo

Bamboo is a part of the Atlassian product suite, and similarly to other tools, it offers building, testing and deploying code and supports numerous languages. It has strong integrations to other Atlassian products that are relevant for the CI cycle, such as JIRA and Bitbucket.

What does it do?

Build, test and deploy are all part of Bamboo’s package, and the testing part is done with the help of a Bamboo Agents. Similar to the agents in Java monitoring, Bamboo also offers two types; the local agents run as part of the Bamboo server as part of its process, while the remote agents run on other servers and computers. Each agent is assigned to the builds that match its capabilities, which allows assigning different agents to different builds.

The main advantage Bamboo offers is the strong ties to the rest of Atlassian’s products, such as JIRA and Bitbucket. With Bamboo you can see code changes and JIRA issues that have been introduced into the code since the last deployment. That way, developers can sync their workflow and always stay on track and know which version is next, and what (should) have been fixed.

One more thing: Bamboo comes with Atlassian’s strong back, along with better workflows for the company’s existing products. If you want to add JIRA and Bitbucket to your CI process in a seamless way and are willing to pay for it, Bamboo is worth a try.

Bottom line: Bamboo is powerful as long as you’re using it with Bitbucket and JIRA, and are willing to pay for your CI solution.

bamboo dashboard
The Bamboo dashboard

Final thoughts

The need for faster and shorter release cycles leads to teams having to find tools and workflows that will support new methods for delivering software. Pushing to production on a weekly or even daily or hourly basis also means introducing new errors into production. Now that you’ve moved to a CI/CD workflow, the next step is understanding what’s the missing link in a complete CI/CD toolchain, and how can you add it to your workflow.

Henn is a marketing manager at OverOps covering topics related to Java, Scala and everything in between. She is a lover of gadgets, apps, technology and tea.

Troubleshooting Apache Spark Applications with OverOps OverOps’ ability to detect precisely why something broke and to see variable state is invaluable in a distributed compute environment.
Troubleshooting Apache Spark Applications with OverOps

Next Article

The Fastest Way to Why.

Eliminate the detective work of searching logs for the Cause of critical issues. Resolve issues in minutes.
Learn More