TL;DR: Caching the Library folder can significantly speed up your Unity builds in CI/CD environments. Codemagic CI/CD also allows you to cache the Unity version of your project so that you don’t need to re-download it each time.
This post is written by Mina Pêcheux
When you work on complex Unity projects and start adding more and more resources, you’ll quickly notice that the build times in your Codemagic workflow grow as well. And it can be pretty annoying to realize that a good chunk of the build time is usually devoted to restoring the env and, in particular, the libraries and packages… even if those are exactly identical each time!
Generally speaking, on a local computer, we avoid this issue because after the project has been built the first time, it has cached some assets and data in your Unity project folder to reuse in the future and speed up this process. But can we use caching on a remote Codemagic machine?
The answer is, luckily, yes.
And it’s worth it.
So let’s have a look at how we can leverage this nice Codemagic tool for our Unity projects. :)
Learn why you should use a CI/CD for your Unity projects from this article.
In this article, we are going to:
- Quickly explore the fundamentals of caching for a Unity project
- Learn why removing unused packages is important
- Discover how to set up a Codemagic workflow with caching
- Study a useful trick for caching an entire Unity version on a Codemagic machine
Note: If you already know about the Unity cache system, or if you just want to see how to use it with Codemagic, you can skip to the third section.
Prerequisites
For this article, we will assume that you are already familiar with how to create a sample Unity project, version it as a Git repo, and link it to Codemagic by creating a new associated app. If you need a reminder on how to perform all these steps, feel free to have a look at this other article.
If you want to read ahead and look through the final sample project, you can find it on GitLab over here. 🚀
You will also need a Unity Plus or Pro license that can be activated via the command line in the Codemagic workflow.
In case you are not a Codemagic user yet, you can sign up here:
A quick tour of Unity’s caching
A Unity project consists of a lot of files, some being your own assets and others being autogenerated logs, temporary data, and references for look-up tricks. The fact that Unity can reconstruct these non-user resources is why, when you version a Unity project, you typically ignore all folders except for Assets, Packages, and ProjectSettings.
Among those autogenerated files, the ones that are crucial to caching are those inside of Library. Basically, this is where Unity pulls and stores all the external packages your app requires and that are listed in the manifest.json file inside of Packages. This setup is a very common one: The Package Manager tool keeps this list of necessary dependencies up to date behind the scenes, and when you share your project, you share this list but not the actual contents of the required packages. Then, any other Unity executable on another machine can reread this list and refetch the dependencies.
Those files are the ones worth caching because they take quite some time to restore (since the computer has to download the entire list), and they usually don’t change that often once your project is set up and ready to build. Chances are that when the time comes to export your game, you’ll know what packages you need, and you won’t update this list — at least until the next big feature comes in.
To see how the autogenerated folders and, particularly, the Library folder impact the build time, we can run a series of tests in which we either keep or remove those directories before building and compare the results. In a very simple project like our sample one 🚀, when we run each test 10 times to alleviate randomness, we can get something like:
Build time (without caching): 70.8 sec
Build time (with caching): 23.5 sec
We clearly see that keeping those folders is a great way to reduce the build time!
Removing unused packages
Another important thing to keep in mind is that, by default, many Unity templates preinstall a lot of dependencies… even if your project doesn’t actually need them! So, when you’re building and shipping your game, remember to check whether your package manager currently lists libraries you don’t use.
For example, in the same sample project, removing all the (unused) 2D-related packages significantly decreases the build time:
Build time (without caching, with 2D): 70.8 sec
Build time (without caching, no 2D): 40.8 sec
You might think that this is only interesting if you don’t use caching, but if you try the same comparison while keeping the cache, you’ll notice that there is a difference in this case too:
Build time (with caching, with 2D): 23.5 sec
Build time (with caching, no 2D): 16.0 sec
That’s because Unity still has to read and “register” the packages when it starts the build, so whether the package is actually used or not it will still impact the initialization time a bit.
Setting up a Codemagic workflow with Unity build cache
Codemagic has a built-in cache system that is very easy to use: You just need to use the cache and cache_paths keywords to tell the tool which directories or files you want to keep in memory from one run to another:
cache:
cache_paths:
- a/first/path
- a/second/path
- ...
This cache system operates per workflow, meaning that you can “save” different file paths for the different workflows defined in your codemagic.yaml configuration file.
In our case, as discussed before, we want to cache the Library folder that will be pulled from the Git repo during the initialization steps. This means we need to add the following to our Codemagic workflow setup:
cache:
cache_paths:
- $CM_BUILD_DIR/Library
- ...
Here, CM_BUILD_DIR is a Codemagic built-in environment variable available in any Codemagic workflow run that points to the path where the code was cloned.
Note that the cache path should always be written in this Unix style, even on a Windows computer. For example, in my sample project, which is meant to run on a Windows Codemagic instance, I have the following:
workflows:
unity-win-workflow-cached:
name: Unity Win (With Cache)
environment:
groups:
- unity # <-- (Includes UNITY_SERIAL, UNITY_USERNAME, UNITY_PASSWORD)
cache:
cache_paths:
- $CM_BUILD_DIR/Library
scripts:
- name: Activate license & Build project
script: |
cmd.exe /c "$env:UNITY_HOME\\Unity.exe" -batchmode -quit -logFile -projectPath . -executeMethod BuildScript.BuildWindows -nographics -serial $env:UNITY_SERIAL -username $env:UNITY_USERNAME -password $env:UNITY_PASSWORD
- name: Export Unity
script: |
cd win
7z a -r release.zip ./*
artifacts:
- win/*.zip
publishing:
scripts:
- name: Deactivate license
script: cmd.exe /c "$env:UNITY_HOME\\Unity.exe" -batchmode -quit -returnlicense -nographics
Once you’ve run the workflow with caching once, you’ll see that in your Codemagic app settings, in the Caching tab, you can click the Refresh list button on the right to see a new entry corresponding to your workflow ID and the time you last ran the workflow and updated the cache.
Here, you can also clear the cache (either just one line or the whole cache for all the workflows) to force a full rebuild next time and get your entire new updated environment.
To see the cache in action, you can compare the first build (for which the cache wasn’t computed yet) and the second one (with the cache already prepared).
Without cache (first build):
With cache (second build):
Installing and caching a Unity version
Installing a specific Unity version on a Codemagic machine
Sometimes, your workflow requires a specific Unity version. This can be because you’ve prepared your project in an older editor or just one that is not retro-compatible with the one installed by default on a Codemagic machine (see the docs for a Mac or Windows machine).
Now, to install a given Unity executable on the build machine, you have to know its exact version and version changeset.
If you need a Unity editor that is neither the one preinstalled on the Codemagic machine nor the one you use locally, it can be a bit of a long process to get this info. You have to go to the Unity download archive page, find the version you’re looking for, open its changelog, and scroll to the bottom of the page to get its changeset.
However, as explained in the Codemagic docs, if you want to install the Unity version you’re using locally, then all this info is actually readily available in your project’s files. More precisely, it’s in your ProjectSettings/ProjectVersion.txt config file, which should look something like this:
m_EditorVersion: 2020.3.29f1
m_EditorVersionWithRevision: 2020.3.29f1 (2ff179115da0)
Once you’ve found this version and changeset, you just need to define them as environment variables in your Codemagic app, like UNITY_VERSION and UNITY_VERSION_CHANGESET.
environment:
vars:
UNITY_VERSION: 2020.3.29f1
UNITY_VERSION_CHANGESET: 2ff179115da0
At this point, you’ll have the data you need to fetch and install this specific Unity editor using the Hub CLI.
Before you do anything with this CLI, however, you need to make sure that you already have an active Unity license on your computer. This means that you need to activate said license before you install the new executable with the CLI. That’s why you still need to use the preinstalled Unity version for the license activation and return steps.
To differentiate between this preinstalled executable and the newly downloaded one, you can once again use environment variables in your workflow with the path to both executables (in addition to the usual license-related Unity env vars stored in the unity env group):
- On Mac:
environment:
groups:
- unity
vars:
UNITY_VERSION: 2020.3.29f1
UNITY_VERSION_CHANGESET: 2ff179115da0
UNITY_BIN: $UNITY_HOME/Contents/MacOS/Unity
UNITY_VERSION_BIN: /Applications/Unity/Hub/Editor/${UNITY_VERSION}/Unity.app/Contents/MacOS/Unity
- On Windows:
environment:
groups:
- unity # <-- (Includes UNITY_SERIAL, UNITY_USERNAME, UNITY_PASSWORD)
vars:
UNITY_VERSION: 2020.3.29f1
UNITY_VERSION_CHANGESET: 2ff179115da0
UNITY_BIN: $UNITY_HOME/Unity.exe
UNITY_VERSION_BIN: "C:\\Program Files\\Unity\\Hub\\Editor\\$UNITY_VERSION\\Editor\\Unity.exe"
UNITY_HUB: "C:\\Program Files\\Unity Hub\\Unity Hub.exe"
After you’ve activated your license, all that’s left to do is call a few commands to actually get and install the Unity editor with a specific version:
- On Mac:
- name: Install Unity version
script: |
/Applications/Unity\ Hub.app/Contents/MacOS/Unity\ Hub -- --headless install --version $UNITY_VERSION --changeset $UNITY_VERSION_CHANGESET
/Applications/Unity\ Hub.app/Contents/MacOS/Unity\ Hub -- --headless install-modules --version $UNITY_VERSION -m ios android
- On Windows:
- name: Install Unity version
script: |
New-Item ".\install-unity.bat" #create an empty batch file
Set-Content install-unity.bat "`"$env:UNITY_HUB`" -- --headless install -v $env:UNITY_VERSION --changeset $env:UNITY_VERSION_CHANGESET"
Add-Content install-unity.bat "`"$env:UNITY_HUB`" -- --headless install-modules --version $env:UNITY_VERSION -m ios android"
Start-Process -FilePath ".\install-unity.bat" -Wait -NoNewWindow #start executing the batch file
And there you are! From this point on, you’ll be able to use the UNITY_VERSION_BIN environment variable to call the new executable (i.e., the newly installed Unity version) and complete all your build steps. :)
Caching a Unity version
This is all well and good, but if you take a look at a workflow with this Unity installation step, you’ll quickly notice that it is quite time consuming. Since the machine has to go online, get the executable, download it, and then install it, it can easily take several minutes and actually take up most of your “build time.”
Once again, caching comes to the rescue here!
A “Unity version” is, in truth, just a folder on the computer that is in a specific location. This means that we can use a cache path as we did before, except this time, we’ll target a directory inside the Unity Hub installation folders.
This is fairly straightforward — on a Mac, for example, you can just add this line in your caching configuration:
cache:
cache_paths:
- $CM_BUILD_DIR/Library
- /Applications/Unity/Hub/Editor/$UNITY_VERSION # <-- cache Unity version (on Mac)
On Windows, the folder would be:
C:\Program Files\Unity\Hub\Editor\$UNITY_VERSION
This will tell the Codemagic machine to only install the new Unity editor at this location if it is not there already to avoid re-downloading the exact same content every time.
If you run this new workflow (at least twice to get the cache), you’ll notice a couple of things. Let’s take my own trials as an example:
Without cache:
With cache:
- The difference is pretty clear: We do avoid repeating the installation step, which took about 11 minutes the first time
- On the other hand, the Cleaning up step was slightly longer than usual because we needed to package and save the full Unity editor, which requires about 4 minutes
- All in all, this caching reduces the total build time by 10 minutes!
We can see that caching a Unity version is very valuable and can significantly reduce the duration of our Codemagic workflow. However, it does result in a larger cache, which can cause some performance issues. If you go back to your cache page after running this kind of pipeline, you’ll see that Codemagic displays a warning and highlights your heavy cache.
As a reference, here is the final full workflow for the Mac cached version (feel free to have a look at the GitLab 🚀 for the Windows sample):
workflows:
unity-mac-install-workflow-cached:
name: Unity Install Cached Mac
environment:
groups:
- unity
vars:
UNITY_VERSION: 2020.3.29f1
UNITY_VERSION_CHANGESET: 2ff179115da0
UNITY_BIN: $UNITY_HOME/Contents/MacOS/Unity
UNITY_VERSION_BIN: /Applications/Unity/Hub/Editor/${UNITY_VERSION}/Unity.app/Contents/MacOS/Unity
cache:
cache_paths:
- $CM_BUILD_DIR/Library
- /Applications/Unity/Hub/Editor/${UNITY_VERSION}
scripts:
- name: Activate License
script: $UNITY_BIN -batchmode -quit -logFile -serial ${UNITY_SERIAL?} -username ${UNITY_USERNAME?} -password ${UNITY_PASSWORD?}
- name: Install Unity version
script: |
/Applications/Unity\ Hub.app/Contents/MacOS/Unity\ Hub -- --headless install --version $UNITY_VERSION --changeset $UNITY_VERSION_CHANGESET
- name: Build
script: $UNITY_VERSION_BIN -batchmode -quit -logFile -projectPath . -executeMethod BuildScript.BuildMac -nographics
artifacts:
- mac/*.app
publishing:
scripts:
- name: Deactivate License
script: $UNITY_BIN -batchmode -quit -returnlicense -nographics
Conclusion
A Unity project is a combination of many assets — the text files, images, 3D models, and metadata that make up your game. But some of them are not handmade: They are autogenerated by Unity if not readily available.
This logic is nice, but it also requires some computing power and time. If the files to regenerate are the same each time, then it can be interesting to use caching to keep them in between your build runs. Luckily, Codemagic has an easy-to-use caching system that makes it simple to store a folder of dependencies or even an entire Unity version. :)
I hope you enjoyed this tutorial. Of course, don’t hesitate to share your ideas for other DevOps topics you’d like me to make Unity tutorials on! You can find me on Codemagic’s Slack, on Medium, or on Twitter.
You can check out the sample project on GitLab over here. 🚀
