Martin Remmelgas, CEO
On September 3, we changed how CPU is allocated to the virtual machines that run your builds on our Mac mini machines. The change was meant to fix machine availability issues by making our platform more stable and, as a side effect, build times more consistent. Instead, many workflows became slower, most noticeably on Mac mini M2 machines.
We didn’t announce this change at the time because we didn’t expect it to affect your builds. We also didn’t catch its impact before rolling it out, and we didn’t notice it until now.
We’re extremely sorry for the slower builds many of you experienced in the weeks that followed, especially since fast builds are one of the reasons many of you choose Codemagic.
We have since reverted the change and are working on a better solution to the underlying issue. Below, we explain what we changed and why, what went wrong, and what we’re doing to fix it.
What we changed and why
Each of our Mac mini hosts runs two virtual machines side by side, and until September 3, the two VMs shared the host’s CPU dynamically, so a VM could use more CPU when its neighbor was less busy.
During peak usage, this setup caused problems with our platform: particularly demanding builds could affect our ability to start new builds, so some builds waited in the queue longer than they should have.
To address this, on September 3 we capped the CPU each VM can use. The change applied to both our Mac mini M2 and M4 machines.
What we expected
In addition to improved platform stability, we expected a more consistent build experience. Sharing CPU between VMs lets a build use spare capacity when its neighbor is less busy, but it also means build times can vary somewhat from run to run.
Before rolling the change out, we tested it and saw no significant impact on build durations. Based on those results, we were confident it would make build times more predictable without slowing builds down.
What actually happened
Our tests didn’t show the full picture. We failed to predict how the cap would affect builds that need more CPU, and in hindsight it’s clear that many workflows slowed down as a result.
Comparing the week before and the week after the change, around three in four workflows running on Mac mini M2 machines became slower, and more than half slowed down by over 10%. For a typical M2 workflow, build time increased by about 13%, or roughly a minute per build. Not every workflow was affected the same way. Around one in four M2 workflows didn’t slow down, and about one in ten actually ran more than 10% faster after the change.
The impact depended on how long a workflow usually takes:
| Typical build time before the change | Workflows that became slower | Typical increase |
|---|---|---|
| Under 10 minutes | About 4 in 5 | About 13%, around 35 seconds |
| 10 to 30 minutes | About 3 in 4 | About 14%, around 2 minutes |
| Over 30 minutes | Just over half | About 4%, around 2 minutes |
On Mac mini M4 machines, where demand on shared resources is lower, the impact was smaller: a typical workflow ran around 5% slower.
What we’re doing about it
On October 23, we reverted the change. We know the fix we rolled out on September 3 was not the right one, and we’ll be much more careful with any further changes. Before rolling out anything that affects how resources are allocated to your builds, we’ll test it far more thoroughly, including with the kinds of CPU-heavy builds our earlier tests missed.
If you’re on a metered plan, you may have paid for extra build time because of this change. If so, we’ll contact you and compensate you by applying a credit to your next invoice, so you don’t need to do anything.
More broadly, we’re continuously upgrading our Mac pools and adding more machines as demand grows, and we’re investing in our platform to make builds more stable and faster.
Finally, we are going to improve our monitoring so we can spot any impact on your build speeds much sooner. It took us far too long to notice it this time.
In summary
We’re truly sorry for the trouble this change caused. We wanted to be open about what happened, even though the change didn’t work out as we intended. If you have any questions or concerns, please reach out to us.
