On the Hunt for the Fastest CI Service

Imagine that you’re a developer making changes to a shared codebase. Following git etiquette, you create a new branch for the changes to live in and make your changes there. But, instead of merging those changes immediately, you keep your code in your local branch until you have finished a larger set of related changes. Unfortunately, when you go to merge your branch into the master branch, another developer has already merged into the master branch, creating merge conflicts. Now, you are spending hours making sure you don’t break anything while resolving the conflicts: welcome to integration hell.

Until relatively recently, every developer had experienced integration hell. Then, in 1991, a man named Grady Booch proposed Continuous Integration (CI) – the process of integrating code into a shared repository frequently so as to minimize the pain of dealing with conflicting changes. As an added bonus, continuously integrated development allows us to verify each integration with an automated build and automated tests, and when using either Continuous Deployment or Continuous Delivery, the changes can be deployed to production as soon as they are integrated.

CI platforms exist to automate these bonuses. A CI platform can test, build, and deploy the code without any user intervention (though many still prefer to do that last step manually). When using a good CI platform, developers can deploy bug fixes quickly and introduce new features seamlessly. When using a bad CI platform, developers may spend hours waiting for their changes to be processed, only to receive an error in a test that should have been thrown immediately.

Here at Babbel, we have used Travis as our CI platform for years. With changing climates (see #travisAlums on Twitter), we decided to research other options. Below, I’ve ranked Travis against two other CI platforms: CircleCI – a popular alternative used in a number of large projects including the open source Vue.js (available on GitHub) – and Semaphore – the self-labeled “fastest CI/CD platform”; we will soon see if the latter claim is true.

Obviously, speed will be a key metric in our comparison. However, we will also note the usability of each platform. To measure build speed, I configured all three services on both an internal Babbel codebase and Vue.js. Please note that both of these codebases are JavaScript-based; you may find different results using different languages. We executed each app’s production build process thirty times on each codebase using each CI service and recorded the self-reported times. Note that the data is self-reported, and while it appears that Circle and Semaphore report the “start” time as the time a commit is received, Travis reports when the task exits its initial queue, which seems to be roughly thirty seconds after the other two start.

In terms of speed, Travis ranked third in our tests. Despite not reporting the time a build spends in its queue, Travis’ reported build times lagged behind both Circle and Semaphore, and Travis’ fastest average build speed took 60 percent more time than the next slowest! Specifically, there is only one data point in which Travis outperforms any other service.

The difference in speed between Circle and Semaphore was less glaring. Circle took only 13 and 43 percent more time than Semaphore on our code and the Vue codebase respectively. A two-sample t-test, which measures the difference between any two numerical distributions (calculated from evanmiller.org), confirms Semaphore as the faster alternative in both cases. For statistics nerds: using 99% confidence intervals and the null hypothesis of equality between distributions, we achieved p values of 0.0015 and 0.0001 on our code and Vue, respectively.

When considering usability, Travis lacked features found in both of the other services. With both Circle and Semaphore, starting a new project is as simple as saving an autogenerated file into your GitHub repository, but getting started with Travis involves wrapping your head around the specifics of the configuration files and weeding through the unintuitive documentation. Similarly, both Circle and Semaphore support parallelization naturally using their workflows and pipelines, but enabling the same functionality with Travis involves manually using their build matrices, in which you manually configure a number of virtual machines.

Usability is clearly improved for Circle and Semaphore. They both have incredibly simple tools to create an initial configuration file, and both offer a fully functional CLI for those so inclined. Semaphore offers a fairly minimalist dashboard, but still supports core functionality (though before their recent redesign, functionality as simple as deleting projects wasn’t possible from the dashboard). Circle, in contrast, has a beautiful and intuitive online dashboard as well as a similarly powerful CLI, and provides a variety of views as well as insights into build statistics, shown above. Additionally, Circle has by far the most readable and thorough documentation. They even have a whole section of documentation dedicated to migrating from Travis, which provides better explanations of Travis’ commands than their own documentation.

The choice between Semaphore and Circle is one of personal preference: would you sacrifice a small increase in build time for a better developer experience? Personally, I use Circle for my own projects–I find the dashboard intuitive and I enjoy looking at Circle’s “insights”. Based on the results of my research, the fastest choice appears to be Semaphore, and most usable appears to be Circle. But between the easy-to-use Circle and the speedy Semaphore, it’s up to you.

Appendix:

Photo by Jeremy Straub on Unsplash