Babbel Bytes

Insights from the Babbel engineering team

On the Hunt for the Fastest CI Service

Neil Natarajan

A speed- and usability-focused comparison of the Travis CI, CircleCI, and Semaphore CI Continuous Integration services.


Imagine that you’re a developer making changes to a shared codebase. Following git etiquette, you create a new branch for the changes to live in and make your changes there. But, instead of merging those changes immediately, you keep your code in your local branch until you have finished a larger set of related changes. Unfortunately, when you go to merge your branch into the master branch, another developer has already merged into the master branch, creating merge conflicts. Now, you are spending hours making sure you don’t break anything while resolving the conflicts: welcome to integration hell.

Until relatively recently, every developer had experienced integration hell. Then, in 1991, a man named Grady Booch proposed Continuous Integration (CI) - the process of integrating code into a shared repository frequently so as to minimize the pain of dealing with conflicting changes. As an added bonus, continuously integrated development allows us to verify each integration with an automated build and automated tests, and when using either Continuous Deployment or Continuous Delivery, the changes can be deployed to production as soon as they are integrated.

CI platforms exist to automate these bonuses. A CI platform can test, build, and deploy the code without any user intervention (though many still prefer to do that last step manually). When using a good CI platform, developers can deploy bug fixes quickly and introduce new features seamlessly. When using a bad CI platform, developers may spend hours waiting for their changes to be processed, only to receive an error in a test that should have been thrown immediately.

Here at Babbel, we have used Travis as our CI platform for years. With changing climates (see #travisAlums on Twitter), we decided to research other options. Below, I’ve ranked Travis against two other CI platforms: CircleCI - a popular alternative used in a number of large projects including the open source Vue.js (available on GitHub) - and Semaphore - the self-labeled “fastest CI/CD platform”; we will soon see if the latter claim is true.

Obviously, speed will be a key metric in our comparison. However, we will also note the usability of each platform. To measure build speed, I configured all three services on both an internal Babbel codebase and Vue.js. Please note that both of these codebases are JavaScript-based; you may find different results using different languages. We executed each app’s production build process thirty times on each codebase using each CI service and recorded the self-reported times. Note that the data is self-reported, and while it appears that Circle and Semaphore report the “start” time as the time a commit is received, Travis reports when the task exits its initial queue, which seems to be roughly thirty seconds after the other two start.


CI Services on Babbel Magazine

Measurement of Build Speed of CI Services (see Appendix for raw data)


CI Services on Vue

Measurement of Build Speed of CI Services (see Appendix for raw data)


In terms of speed, Travis ranked third in our tests. Despite not reporting the time a build spends in its queue, Travis’ reported build times lagged behind both Circle and Semaphore, and Travis’ fastest average build speed took 60 percent more time than the next slowest! Specifically, there is only one data point in which Travis outperforms any other service.

The difference in speed between Circle and Semaphore was less glaring. Circle took only 13 and 43 percent more time than Semaphore on our code and the Vue codebase respectively. A two-sample t-test, which measures the difference between any two numerical distributions (calculated from evanmiller.org), confirms Semaphore as the faster alternative in both cases. For statistics nerds: using 99% confidence intervals and the null hypothesis of equality between distributions, we achieved p values of 0.0015 and 0.0001 on our code and Vue, respectively.

When considering usability, Travis lacked features found in both of the other services. With both Circle and Semaphore, starting a new project is as simple as saving an autogenerated file into your GitHub repository, but getting started with Travis involves wrapping your head around the specifics of the configuration files and weeding through the unintuitive documentation. Similarly, both Circle and Semaphore support parallelization naturally using their workflows and pipelines, but enabling the same functionality with Travis involves manually using their build matrices, in which you manually configure a number of virtual machines.


Screenshot from CircleCI UI

Insights from Circle's Dashboard

Usability is clearly improved for Circle and Semaphore. They both have incredibly simple tools to create an initial configuration file, and both offer a fully functional CLI for those so inclined. Semaphore offers a fairly minimalist dashboard, but still supports core functionality (though before their recent redesign, functionality as simple as deleting projects wasn’t possible from the dashboard). Circle, in contrast, has a beautiful and intuitive online dashboard as well as a similarly powerful CLI, and provides a variety of views as well as insights into build statistics, shown above. Additionally, Circle has by far the most readable and thorough documentation. They even have a whole section of documentation dedicated to migrating from Travis, which provides better explanations of Travis’ commands than their own documentation.


Overall Comparison
Travis Circle Semaphore
Performance
Median Speed on Babbel Magazine (h:mm:ss) 0:02:33 0:01:31 0:01:19
Median Speed on Vue (h:mm:ss) 0:03:55 0:02:02 0:01:29
Executes in
MacOS VM Yes Yes Yes
Linux VM Yes - Ubuntu Only Yes - any from Amazon Machine Images Yes - Ubuntu Only
Windows VM Yes - Very Early Support Yes No
Docker Containers No Yes No
Deploying with Docker
Docker Commands Yes Yes Yes
Docker Compose Yes Yes - Only on VMs Yes
Layer Caching Yes Yes - On all but Windows Yes
Terraform Support
Official Terraform Provider No No No


The choice between Semaphore and Circle is one of personal preference: would you sacrifice a small increase in build time for a better developer experience? Personally, I use Circle for my own projects–I find the dashboard intuitive and I enjoy looking at Circle’s “insights”. Based on the results of my research, the fastest choice appears to be Semaphore, and most usable appears to be Circle. But between the easy-to-use Circle and the speedy Semaphore, it’s up to you.


Appendix:


Raw Build Speed Data (h:mm:ss)
Travis Circle Semaphore
Babbel Magazine
Trial 0 0:02:29 0:01:37 0:01:25
Trial 1 0:02:32 0:01:34 0:01:21
Trial 2 0:02:33 0:01:40 0:01:18
Trial 3 0:02:39 0:01:42 0:01:20
Trial 4 0:02:33 0:01:43 0:01:15
Trial 5 0:02:34 0:01:36 0:01:25
Trial 6 0:02:34 0:01:52 0:01:30
Trial 7 0:02:30 0:01:21 0:01:25
Trial 8 0:02:33 0:01:38 0:01:29
Trial 9 0:02:34 0:01:36 0:01:15
Trial 10 0:02:33 0:01:21 0:01:56
Trial 11 0:02:32 0:01:19 0:01:10
Trial 12 0:02:31 0:01:21 0:01:14
Trial 13 0:02:33 0:01:32 0:01:12
Trial 14 0:02:34 0:01:20 0:01:16
Trial 15 0:02:36 0:01:42 0:01:15
Trial 16 0:02:31 0:01:19 0:01:08
Trial 17 0:02:34 0:01:26 0:01:20
Trial 18 0:02:35 0:01:31 0:01:15
Trial 19 0:02:30 0:01:24 0:01:09
Trial 20 0:02:30 0:01:22 0:01:10
Trial 21 0:02:36 0:01:30 0:01:24
Trial 22 0:02:35 0:01:21 0:01:31
Trial 23 0:02:43 0:01:51 0:01:29
Trial 24 0:02:32 0:01:20 0:01:16
Trial 25 0:02:36 0:01:29 0:01:25
Trial 26 0:02:37 0:01:57 0:02:12
Trial 27 0:02:32 0:01:29 0:01:19
Trial 28 0:02:29 0:01:26 0:01:09
Trial 29 0:02:34 0:01:44 0:01:12
Arithmetic Mean 0:02:33 0:01:32 0:01:22
Geometric Mean 0:02:33 0:01:32 0:01:21
Median 0:02:33 0:01:31 0:01:19
Vue
Trial 0 0:03:51 0:01:47 0:01:27
Trial 1 0:03:57 0:02:13 0:01:17
Trial 2 0:03:55 0:01:56 0:01:19
Trial 3 0:04:01 0:02:09 0:01:20
Trial 4 0:04:12 0:01:57 0:01:32
Trial 5 0:04:02 0:01:44 0:01:27
Trial 6 0:03:57 0:01:58 0:01:24
Trial 7 0:04:11 0:02:10 0:01:32
Trial 8 0:03:55 0:01:52 0:01:21
Trial 9 0:04:02 0:01:58 0:01:31
Trial 10 0:03:48 0:01:56 0:01:34
Trial 11 0:03:44 0:05:28 0:01:31
Trial 12 0:03:50 0:01:45 0:01:20
Trial 13 0:04:11 0:02:12 0:01:23
Trial 14 0:03:54 0:01:49 0:01:31
Trial 15 0:03:54 0:02:07 0:01:35
Trial 16 0:04:06 0:02:11 0:01:54
Trial 17 0:04:00 0:02:00 0:01:18
Trial 18 0:03:47 0:02:01 0:01:18
Trial 19 0:04:09 0:02:05 0:01:21
Trial 20 0:03:48 0:02:18 0:01:31
Trial 21 0:03:55 0:02:15 0:01:24
Trial 22 0:03:49 0:02:04 0:01:35
Trial 23 0:03:52 0:01:51 0:01:32
Trial 24 0:03:49 0:02:09 0:01:26
Trial 25 0:03:55 0:01:47 0:01:40
Trial 26 0:03:48 0:02:45 0:02:19
Trial 27 0:03:59 0:01:54 0:01:18
Trial 28 0:04:02 0:02:08 0:01:36
Trial 29 0:04:01 0:02:02 0:01:35
Arithmetic Mean 0:03:57 0:02:09 0:01:30
Geometric Mean 0:03:57 0:02:06 0:01:29
Median 0:03:55 0:02:02 0:01:29
Facebook Twitter Google+ Reddit EMail
Comments