Codemagic has recently decreased its prices thanks to Apple M1 machines. How is it possible for us to provide Apple M1 VMs to everybody, including those on a Free plan, and lower the prices at the same time?
Codemagic’s CTO Mikhail Tokarev took some time to share the details, including the technical aspects behind our recent changes. Here, we’ll explain what it takes to operate a CI/CD service and the technical challenges our dev team had to overcome to make it possible for us to provide our services at lower prices.
How we use our own Virtual Machine Manager for CI/CD
As a CI/CD service, we create and manipulate hundreds of virtual machines every minute to provide a secure and standardized environment to execute users’ tasks. This requires a unified approach to managing virtual machines running macOS, Linux, or Windows and working with different virtualization technologies.
To address these challenges, we have implemented a dedicated service that we have named Virtual Machine Manager (or VMM for short). VMM is the abstract layer that orchestrates all virtual machines. It has the following functionalities:
- Acquires/releases virtual machines with required parameters, such as the instance type, operating system, and software version.
- Forces the wiping of machines and restores them to the user pool after the job is completed or if the time limit is exceeded.
- Runs sanity checks and controls machine availability. (When you host over 100 different pieces of hardware, you face network issues, power outages, and dying storage drives on an almost daily basis.)
Using VMM allows us to hide underlying implementations such as virtualization software and easily add new hardware to the technology stack we provide to our users.
In this article, I want to describe our experience of adding new Apple M1 machines and later migrating to new virtualization software with zero downtime, a change that went almost unnoticed by end users.
Why macOS virtualization is hard
While there are many cloud services that provide Linux and Windows instances, working with Apple macOS is still somewhat challenging. This is primarily because of the following two problems:
- Due to Apple’s license policy, you can only run macOS on Apple hardware. And because of the specific form factors that Apple hardware comes in, not all data centers can allocate it. In addition to providing required service-level agreements, data centers need to have standby machines to quickly replace those that went down.
- As of 2022, there is still no production-ready way to use Docker-like solutions to run macOS inside a container. You need to use other virtualization software like VMware ESXi to provide reasonable performance, hardware acceleration, and support for nested virtualization. Unfortunately, the previous generations of Apple computers are not supported by ESXi, and VMware announced that ESXi 7 will be the final release supporting Apple Mac platforms in general.
At Codemagic, we historically used ESXi as a bare-metal hypervisor and VMM to manage virtual machines.
Life after ESXi
In 2017, Apple announced a new generation of computers using T2 security chips. The T2 is integral in securely powering up and the boot sequence, and it does not allow other operating systems like ESXi to be used.
Starting from macOS 11.0, Apple released a new Virtualization framework that provides high-level APIs for creating and managing virtual machines. This provided an opportunity to use the latest and most powerful Apple computers for virtualization.
Since we’ve tried building apps on Apple silicon M1 processors and seen their performance and how much they can speed up builds, we wanted to introduce Apple silicon build machines to Codemagic users. Since we couldn’t use ESXi for M1s, we had to come up with some other solution that relied on Apple’s Virtualization framework.
Attempt #1: Veertu Anka for macOS virtualization on Apple silicon
We decided to use Apple M1 Mac minis powered by silicon chips and Anka v3 to provide the greatest performance experience to our users. Since we already had experience integrating VMM with new platforms (initially, we launched Linux machines backed by Google Cloud Platform, followed by Windows machines later in 2021), the development part was really straightforward.
The Anka CLI tools have everything you could need and were originally designed to be used within CI/CD services like Codemagic.
However, Anka is only a wrapper for the Apple Virtualization API. Even though it provides Anka Controller to manage build nodes and Anka Registry to manage images, our VMM solution is more universal and better integrated into our infrastructure, so we didn’t use many of the features Anka provides. Since Anka’s license was very expensive and we were only utilizing around 10% of its functionality, we were constantly thinking about how we could improve our infrastructure and lower our costs.
Attempt #2: Emerging favorite: Tart for Apple silicon
Tart is a relatively new tool for running and managing virtual machines on Apple silicon. (At the time of writing, the latest version of Tart is 0.30.) The CLI tools were also built with the idea of using Tart in automated solutions, so adding and trying it was the natural next step.
Compared with the Anka CLI tools, Tart has fewer commands available and does not have built-in tools for interacting with the VM (e.g., copying files to/from a running VM or executing commands inside a running VM). But all this can be achieved by using SSH, for example. Also, instead of the proprietary image registry used by Anka, Tart supports OCI registries, which allowed us to migrate to a self-hosted Docker registry for faster provisioning of new Apple M1 machines.
Another different feature we discovered (not related to virtualization but actively used by our users) was the ability to redirect inbound network traffic inside the VM. Anka provides this feature out of the box, but for Tart, we had to implement our own solution. Luckily, macOS has the pfctl utility for controlling network traffic (similar to iptables in Linux), which makes it simpler to implement.
Here are our conclusions after using Tart in production for two months:
- Tart is still in active development and you can encounter some issues, but the team behind Tart has been awesome so far and instantly fixed reported issues.
- Thanks to the minimal overhead of using the Apple Virtualization API, we’ve seen some performance improvements in booting new virtual machines compared with Anka.
- The main difference, of course, is in the costs. Tart is an open-source project, while Anka is not — it has quite an expensive license. So, getting rid of that expensive license by migrating from Anka to Tart has allowed us to provide Apple M1 machines at half the original price.
Conclusion
Codemagic has managed to optimize its platform to provide more build machines to our users with lower infrastructure costs, which means that we can offer lower prices to our users as well (especially for M1 machines).
To conclude, here’s how Codemagic’s pricing has changed:
- Codemagic’s M1 Mac mini machines are now half the price.
- With our new Pay-as-you-go plan, users will never pay more than $299/month — no matter how many build minutes and team seats they use. This plan also includes up to three concurrencies.
- Users no longer have to pay for team seats.
We hope that with our new price changes, more users will be able to afford to use Codemagic’s fastest build machines and continue using our service without worrying about getting surprisingly high bills at the end of the month.
