You can read a lot about why continuous integration is a powerful technique and how it can speed up your mobile app development process. We decided to dig deeper and analyze the use cases of a continuous integration tool tailored for Flutter.
We gathered data from Codemagic, a dedicated CI/CD tool for Flutter, and analysed more than 80k builds starting from the birth of Codemagic in December 2018. Here are the results in real numbers.
Let’s see how Flutter apps are doing on Codemagic CI/CD servers, how implementing CI/CD benefits our users and what you can take away from our data.
*PS! Special thanks to our data and machine learning scientist Lena for collecting data and helping us make some sense out of it!
CI/CD for Flutter
Before jumping into numbers, let’s briefly go over what is CI/CD and why Flutter needs a dedicated CI/CD tool.
Continuous integration (CI) is a software development practice of merging developer build copies daily, if not multiple times a day, into a shared code repository. Each integration is verified by an automated build to detect errors and get to the root of the problem as soon as possible without losing track of the development process.
Continuous delivery (CD) is all about the ability to continuously deliver integrated code, be it bug fixes or new features. With Codemagic, you can automate the entire release pipeline and easily distribute your “green builds” for testing or publishing.
The core benefit of Flutter is the ability to develop apps for both iOS and Android, and why not even for the web as the third platform, from a single code base, which enables you to save both development time and resources. And this is also what you would expect from your continuous integration and delivery tool.
As a dedicated CI/CD tool for Flutter, Codemagic enables you to set up your project just once and automate the builds for all platforms, that is Android, iOS and the web. You can also run automated tests on the app or even widgets to get fast results and keep your team in the loop with email or Slack publishing.
The entire workflow from project setup to publishing to app stores has been designed with Flutter apps and Flutter app developers in mind. For example, you can conveniently try a different Flutter or Xcode version via the UI, and build, sign and publish iOS apps without a Mac. And when there’s a new Flutter or Xcode release out, Codemagic will be the first to support it.
Last but not least, Codemagic also runs a Slack community for all Codemagic users (there are already over 900 members!) where you can discuss Flutter and CI/CD with fellow Flutter enthusiasts and talk to our team!
That said, it’s clear that implementing CI/CD into your Flutter development process is crucial for every healthy app, and choosing a tool that enables you to work smartly and effectively is equally important. If you want to know more about why you need a dedicated CI/CD tool for Flutter, take a look here.
CI/CD for Flutter: What developers think?
In September 2018, we offered a special deal for early adopters — FREE CI/CD for Flutter projects in exchange for product feedback. Codemagic was built based on that feedback and that’s one of the reasons why Codemagic has grown into the community-favorite CI/CD tool in 6 months. As a reminder, this is the value Flutter devs see in CI/CD.
“CI for Flutter is even more important than in native development, as there are two different platforms that need to be supported and distributing on both is totally different”
“As Flutter is so fast for the developers to get the app done, the packaging and releasing should be as easy and fast. So for me, CI is very much important for as a Mobile Developer.”
“Continuous Integration is very important because it’s one of the big improvements over react native and we all know now every strong project requires CI/CD”
“CI would improve my productivity as without it I should check everything manually (for example that my project is compatible with the latest version of Flutter)”
“It’s just we don’t want to do any manual work to build apk/ipa and push them to the stores - we want to focus on delivering features. Also it’s important to release fast to make sure that testers use latest builds and not double reporting already fixed issues. And clients are happy to see progress more often. Later on we plan to add tests and we need to make sure that we have a CI in place to leverage on having them
“The main benefit for me (in using CI/CD for my Flutter projects) is the transparency in a project between each team member”
“CI provides a faster feedback loop, therefore its importance is fundamental”
“It’s [CI] crucial to Flutter development. The whole idea behind Flutter is quick turnaround and agile teams with features like hot reload and restart. CI ensures “order in the chaos.”
But now, let’s dig into the 80k+ Flutter builds that we have run in 6 months.
CI/CD & hosting services
Where do Flutter developers like to keep their repositories?
An online survey conducted by Flutter’s User Experience Research team found that the majority of Flutter developers have some mobile app dev background and stay faithful to their preferred tools and features to build apps.
This also applies to repository hosting services. Similarly to our analysis on Nevercode CI/CD builds, we can see that the most popular cloud solutions among Flutter developers are GitHub, GitLab and Bitbucket.
The most popular hosting service by far is GitHub with 73%, followed by Bitbucket with 13% and GitLab with 11%. These are the services that are fully integrated with Codemagic and can be used for signup. Also, the percentage cohorts with the time the integration was added. The GitHub integration was available from day one, while integration with GitLab was added last in January 2019. However, the fact that two thirds of Flutter devs prefer GitHub is nevertheless very interesting.
The remaining 3% of Flutter projects are added from a custom source. That’s the place to point out that we support all Git-based repositories, also those that require SSH authentication, so you can host your Flutter app wherever you like and set up CI/CD with Codemagic.
Fun fact: The average GitLab user builds more, but Bitbucket and custom repository owners have a higher ratio of automatic builds.
- Github 73%
- Bitbucket 13%
- Gitlab 11%
- Custom 3%
Build platforms
For which platforms are Flutter developers building their apps?
The power of Flutter is to nail two platforms with one stone. So, let’s see how Flutter developers are taking advantage of this.
Analyzing our Flutter users, we can see that the majority of Flutter projects (84%) are indeed built for both iOS and Android. However, for whatever reason, there is a relatively big number of Flutter developers who are developing for a single platform only.
About 9% of Flutter projects are built for Android and 7% only for iOS. It’s possible that these are the users that are taking Flutter for a test spin and do not have a serious Flutter app in production…yet.
We also have some early adopters of Flutter for web (around 1% of all projects), but since Flutter for web is still in preview stage, we’re not surprised by the small number of web builds.
- Android + iOS 84%
- Android 9%
- iOS 7%
- Web less than 1%
Testing
What are Flutter developers' testing habits?
Testing is an integral part of building a healthy app, and it becomes more crucial if you’re building for two platforms. It’s a well-known fact that the earlier in the development lifecycle you catch the bug, the less it will cost you later. Period. And that’s the reason why more than 56% of users run tests on their projects with Codemagic.
Our analysis revealed that there are 15 tests per build on average and projects with tests fail less frequently. To illustrate the point, the average percentage of failed builds for projects with tests is about 26% compared to 60% for projects without tests. Every developer knows that healthy code means better sleep at night and more tests keep your app healthy and end users satisfied. See how to test Flutter apps with Codemagic.
What kinds of tests Flutter devs run?
Flutter has a rich set of testing features, including unit testing, widget testing and integration testing. About half of all Flutter builds (49%) make use of Flutter Test (unit and widget tests), 7% of builds have Flutter Driver (UI) tests set up and, after we added support for it, 23% of builds run static code analysis with Flutter Analyzer.
The official Flutter documentation on testing recommends having “many unit and widget tests, tracked by code coverage, plus enough integration tests to cover all the important use cases.” Our results show that, in general, developers' testing habits are in line with Flutter’s recommendations since unit tests are the most popular type of tests run with builds. Well done! However, we’d love to see some more UI tests being executed to ensure that users do not ship a buggy app to production. We know that writing UI tests can be tricky, so we have prepared a comprehensive post about Flutter UI testing to help you get started.
Used tests types:
- Flutter Analyze 23% of builds after we started to support flutter analyze tests.
- Flutter Test 49%
- Flutter Driver (UI) 7%
Stop build if tests fail
Tests are written to prevent you from publishing a buggy app, and when tests fail, in most cases, it does not make sense to run the build. Before building your app, Codemagic runs the testing step, and you can select to stop the build should any of the tests fail. 64% of apps have this nifty feature enabled, which has saved our users in total about 173,032 minutes which they would have otherwise spent on running a build that is likely to fail as well. Don’t waste your time or build minutes and get actionable feedback on your build fast!
Time to resolve (TTR) issues
Every developer may hit a red build from time to time. This is the CI tool doing its job and telling you to take another look at your code to ensure that the latest changes don’t break your app’s functionality. But you only benefit from CI if you take care of the red builds!
Our data shows that 77% of all issues were resolved, that is, the failing build was eventually followed by a successful build. Half of the issues were resolved in less than 78.2 minutes, while 75% of the issues were resolved in less than 1,201 minutes (20 hours). That’s a pretty good result, which shows that our Flutter developers are making good use of continuous integration and responding to build failure notifications either immediately or on the following day. That’s CI done right!
23% of issues remain unresolved, and there might be multiple reasons as to why builds fail. It could be because of bugs in the app or due to mistakes in project setup or configuration. Make sure to take a look at the top 12 configuration mistakes that will fail your mobile CI build. If you need more help, you are more than welcome to join our responsive Slack community and reach out to the Codemagic team. Let’s sort out the red builds together!
Build time
Builds under 9 minutes
Our analysis of Flutter builds shows that 75% of Flutter apps are built successfully in 9 minutes or less. The average finished build (successful or finished with warning) takes less than 10 minutes from pushing new code to getting the build status report and publishing artifacts for both Android and iOS, which is fast! You can truly build, test and deliver your Flutter apps in record time.
We took a closer look at team apps and individual apps. The average build time for apps built in teams is 11 minutes and we can assume that this is because team apps are more complex. In contrast, the average build duration for solo developers is 6.15 minutes.
iOS apps developed and built without a Mac
All iOS developers know that building iOS apps requires a lot of investment into expensive machines and there’s no escaping from code signing and joining the Apple Developer Program to test your app on real devices or publish it to the public via App Store. However, based on the signup data, 65% of our users come from Windows or Linux. We can assume that they don’t use macOS to develop Flutter apps, and thanks to Flutter and Codemagic, they don’t have to! See how you can develop and distribute Flutter apps for iOS without a Mac.
Further investigation reveals that 24% of all successful iOS builds have code signing set up. This means that we have helped to produce an IPA for 13% of all apps and 17% of all teams on Codemagic. And this is something we are extremely proud of! The aim of Flutter is to build Android and iOS apps in record speed and we managed to eliminate some obstacles to accelerate the route to healthy iOS apps. Now, all fast movers can take the maximum out of Flutter.
Saved time
Time is money. And the whole point of having a CI/CD pipeline in place is to automate the grunt work so you could save time and money. Along with the growing popularity of Flutter, we have witnessed tremendous growth in the number of teams and companies adopting Flutter and signing up for Codemagic.
Let’s see how much time CI/CD is saving for Flutter projects.
Since the launch of Codemagic, Flutter developers have run builds for 891,770 minutes in total. When we do our math, we can see that Codemagic saves its users about 50 minutes a day. It’s about 10% of an 8-hour workday. This means you could take a day off after every 8 workdays or come to work one hour later every day. Imagine what you could develop in this extra time!
Conclusions
So, what have we learned by analyzing more than 80k Flutter builds on Codemagic servers?
We learned that the vast majority of developers (84%) build for both Android and iOS and on average have the build artifacts ready in less than 10 minutes. It’s also interesting to keep in mind that you are likely to get more than two times higher success rate on your builds (74%) when you run automated tests with your builds (which 56% of our users do) compared to running builds with no tests present.
Our biggest surprise was the number of users signing up from non-Mac operating systems (65%). Knowing that we have made iOS app development more accessible to a considerable number of developers means a lot to us! Supporting the community with solutions that remove obstacles will definitely remain an important topic for us.
To sum up, you can only win from running builds in a clean CI environment as you’ll both get fast feedback on the health of your code as well as save time. In fact, our analysis revealed that Flutter developers win as much as 50 minutes a day using Codemagic CI/CD.
How are you using the extra time you’re winning by using CI/CD?
