On the Hunt for the Fastest CI Service
A speed- and usability-focused comparison of the Travis CI, CircleCI, and Semaphore CI Continuous Integration services.
Imagine that you’re a developer making changes to a shared codebase. Following git etiquette, you create a new branch for the changes to live in and make your changes there. But, instead of merging those changes immediately, you keep your code in your local branch until you have finished a larger set of related changes. Unfortunately, when you go to merge your branch into the master branch, another developer has already merged into the master branch, creating merge conflicts. Now, you are spending hours making sure you don’t break anything while resolving the conflicts: welcome to integration hell.
Until relatively recently, every developer had experienced integration hell. Then, in 1991, a man named Grady Booch proposed Continuous Integration (CI) - the process of integrating code into a shared repository frequently so as to minimize the pain of dealing with conflicting changes. As an added bonus, continuously integrated development allows us to verify each integration with an automated build and automated tests, and when using either Continuous Deployment or Continuous Delivery, the changes can be deployed to production as soon as they are integrated.
CI platforms exist to automate these bonuses. A CI platform can test, build, and deploy the code without any user intervention (though many still prefer to do that last step manually). When using a good CI platform, developers can deploy bug fixes quickly and introduce new features seamlessly. When using a bad CI platform, developers may spend hours waiting for their changes to be processed, only to receive an error in a test that should have been thrown immediately.
Here at Babbel, we have used Travis as our CI platform for years. With changing climates (see #travisAlums on Twitter), we decided to research other options. Below, I’ve ranked Travis against two other CI platforms: CircleCI - a popular alternative used in a number of large projects including the open source Vue.js (available on GitHub) - and Semaphore - the self-labeled “fastest CI/CD platform”; we will soon see if the latter claim is true.
In terms of speed, Travis ranked third in our tests. Despite not reporting the time a build spends in its queue, Travis’ reported build times lagged behind both Circle and Semaphore, and Travis’ fastest average build speed took 60 percent more time than the next slowest! Specifically, there is only one data point in which Travis outperforms any other service.
The difference in speed between Circle and Semaphore was less glaring. Circle took only 13 and 43 percent more time than Semaphore on our code and the Vue codebase respectively. A two-sample t-test, which measures the difference between any two numerical distributions (calculated from evanmiller.org), confirms Semaphore as the faster alternative in both cases. For statistics nerds: using 99% confidence intervals and the null hypothesis of equality between distributions, we achieved p values of 0.0015 and 0.0001 on our code and Vue, respectively.
When considering usability, Travis lacked features found in both of the other services. With both Circle and Semaphore, starting a new project is as simple as saving an autogenerated file into your GitHub repository, but getting started with Travis involves wrapping your head around the specifics of the configuration files and weeding through the unintuitive documentation. Similarly, both Circle and Semaphore support parallelization naturally using their workflows and pipelines, but enabling the same functionality with Travis involves manually using their build matrices, in which you manually configure a number of virtual machines.
Usability is clearly improved for Circle and Semaphore. They both have incredibly simple tools to create an initial configuration file, and both offer a fully functional CLI for those so inclined. Semaphore offers a fairly minimalist dashboard, but still supports core functionality (though before their recent redesign, functionality as simple as deleting projects wasn’t possible from the dashboard). Circle, in contrast, has a beautiful and intuitive online dashboard as well as a similarly powerful CLI, and provides a variety of views as well as insights into build statistics, shown above. Additionally, Circle has by far the most readable and thorough documentation. They even have a whole section of documentation dedicated to migrating from Travis, which provides better explanations of Travis’ commands than their own documentation.
|Median Speed on Babbel Magazine (h:mm:ss)||0:02:33||0:01:31||0:01:19|
|Median Speed on Vue (h:mm:ss)||0:03:55||0:02:02||0:01:29|
|Linux VM||Yes - Ubuntu Only||Yes - any from Amazon Machine Images||Yes - Ubuntu Only|
|Windows VM||Yes - Very Early Support||Yes||No|
|Deploying with Docker|
|Docker Compose||Yes||Yes - Only on VMs||Yes|
|Layer Caching||Yes||Yes - On all but Windows||Yes|
|Official Terraform Provider||No||No||No|
The choice between Semaphore and Circle is one of personal preference: would you sacrifice a small increase in build time for a better developer experience? Personally, I use Circle for my own projects–I find the dashboard intuitive and I enjoy looking at Circle’s “insights”. Based on the results of my research, the fastest choice appears to be Semaphore, and most usable appears to be Circle. But between the easy-to-use Circle and the speedy Semaphore, it’s up to you.