TL;DR: M1 machines are astonishingly fast! For iOS builds (with caching), we see build time improvements of ~50% compared with the Mac Pros.
CI/CD with M1 machines
In November 2020, Apple released the M1 MacBook and, one year later, the M1 Pro/Max chips. Since then, we have all read and heard from other people about the crazy performance improvements.
So we know that M1 machines are fast, but should we use them in CI/CD to speed up our pipeline? To help answer that question, we compared the M1 mini, Mac Pro, and Mac mini for iOS and macOS builds with Codemagic. In this article, we’ll give you the facts and figures you need to decide if an upgrade is worth it.
This article is written by Nils Reichardt
Comparison setup
Let’s talk about the setup of this comparison and take a look at questions like:
- Which apps have been used for the builds?
- Which platforms?
- Release, debug, or profiling mode?
Apps
To compare the builds of the Flutter apps, we used two apps:
- The default Counter app of Flutter
- Sharezone — an open-source school planner students, parents, and teachers can use to organize together
In the comparison, we focus more on the Sharezone app because Sharezone is a real production app with +300,000 registered users. It uses a bunch of dependencies, like many of the Firebase packages and several other ones (Sharezone is open source — you can find all the dependencies here). Therefore, you will see very realistic build times.
Platforms
We are focusing on the following:
- iOS
- macOS
- Tests (unit and widget tests)
System
We are using Codemagic’s virtual machines for the test, which you can use for CI/CD. These machines have the following configuration:
- Mac Mini: 2,3GHz Quad Core / 8 GB
- Mac Pro: 3.7GHz Quad Core / 32 GB
- M1 Mini: 3.2 Quad Core / 8 GB
Flutter Configuration
Flutter version that we used is 2.10.5
To get a more realistic view, all builds used the release mode of Flutter.
For the tests, we just used the flutter test command.
Dependency caching
We compared the builds without and with caching. For caching, we cached the following paths:
- Dart cache:
$FLUTTER_ROOT/.pub-cache - CocoaPods cache:
HOME/Library/Caches/CocoaPods
For more information about how to use caching in Codemagic, just take a look at the documentation on dependency caching.
Accuracy
We ran every build three times and calculated the average. For example, the later build time for the iOS Counter app is the average time of three builds.
In total, we ran 54 builds. You can also take a look at this spreadsheet, which shows all the raw data and links to the builds.
Results of comparing M1 Mini vs Mac Pro vs Mac Mini
We are coming to the central part of this article: the results. You will see how big the difference is in terms of build time between the Counter and the Sharezone apps. Therefore, it’s essential to use a real app for comparisons like this.
All build times are measured in seconds.
Counter app results
| Platform | Mac mini | Mac Pro | M1 mini |
|---|---|---|---|
| iOS | 92 | 75 | 30 |
| macOS | 89 | 51 | 25 |
Sharezone app results
iOS
| Caching | Mac mini | Mac Pro | M1 mini |
|---|---|---|---|
| No | 967 | 658 | 395 |
| Yes | 892 | 598 | 281 |
The results are incredible! The M1 machines reduce build times (with caching) by 68% compared with the Mac mini and 53% compared with the Mac Pro.
macOS
| Caching | Mac mini | Mac Pro | M1 mini |
|---|---|---|---|
| No | 479 | 466 | 433 |
| Yes | 501 | 401 | 333 |
On macOS, we don’t see as huge of improvements compared with the iOS builds, but the M1 builds (with caching) are still 34% faster than the Mac mini builds and 17% faster than the Mac Pro builds.
Unit and widget tests
| Mac mini | Mac Pro | M1 mini |
|---|---|---|
| 84 | 90 | 95 |
When comparing the machines in terms of unit and widget tests, we see nearly no differences between them.
Side note: Sharezone has 238 unit and widget tests.
Overall
Summary
To sum up the results: The M1 machines are nearly 50% faster than the Mac Pro machines, and their results are especially interesting when it comes to integration tests. We all know that slow CI pipelines suck. However, not only will M1 machines reduce the build times of your integration tests, but they are also the way to go if you want to publish your apps on the App Store or Play Store.
This article is written by Nils Reichardt, Co-Founder of Sharezone, a collaborative school planner for Android, iOS, Web and macOS with +300.000 registered users. He fell in love with Flutter since the first beta release in March 2018. You can find Nils on Twitter, GitHub and LinkedIn.
