Run Flutter integration tests on real devices with AWS Device Farm and Sylph

Sep 23, 2019

Up until now, Codemagic was able to run Flutter Driver tests on Android emulator or iOS simulator. Thanks to an open-source command-line utility called Sylph, we now have out-of-the-box AWS Device Farm integration available on Codemagic so you can easily test your Flutter apps on a vast selection of physical Android and iOS devices as part of the build.

What is Sylph and why do we need it?

AWS Device Farm offers a huge pool of Android and iOS devices for running integration tests on many real devices at once. This is a great option for ensuring your release builds run as expected before pushing the app out to app stores. Prior to Sylph, there was no easy way to use AWS Device Farm for Flutter integration tests since AWS Device Farm supports native and Appium tests only and Flutter Driver tests are not in the list of supported frameworks.

Sylph, developed by Maurice McCabe, opens the door to AWS Device Farm for Flutter apps. Under the hood, Sylph makes it look as if you’re running Appium tests. Sylph can be set up locally on Mac, Linux or Windows using command line, but it also works in CI environments — such as Codemagic. The entire AWS Device Farm test run configuration, including test suites and the devices to be used for testing, is defined in a sylph.yaml file. You can see a configuration example in the Sylph repository.

Sylph and AWS Device Farm on Codemagic

Codemagic has both Sylph and AWS Device Farm integrated into its build pipeline, making it really easy for Flutter developers to use AWS Device Farm for testing their apps. An AWS account and a simple sylph.yaml configuration file in the repo is all that is needed for making these two seamlessly work. And if you don’t have a Sylph configuration file, we will create one during the build to run the tests with default configuration.

Let’s walk through the process of configuring Codemagic to run Flutter Driver tests on real devices with the AWS Device Farm integration.

Step 1. Set up AWS Device Farm

Running integration tests on AWS Device Farm can be enabled in App settings > Test. First, you need to check Enable Flutter Driver tests and then select where to run them. As soon as you select AWS Device Farm, you will see the fields to enter your AWS access key ID and AWS secret access key.

If you’re going to run Flutter Driver tests on Android devices only, you can save the settings and finish the setup here. However, if your device pool contains iOS devices, you will need to set up fastlane match and add the iOS-specific environment variables to Codemagic in App settings > Environment.

Check out our documentation for more details on requirements and configuration.

Step 2. Start the build

If you’re happy with the app settings, click Start new build.

During the build, Codemagic searches the repository for a sylph.yaml file to configure the AWS Device Farm test run and generates a default configuration file if no existing configuration is found. By default, we run the tests in your project_root/test_driver folder on Google Pixel 2 (OS 8.0.0).

Step 3. View the results

While you can monitor the build logs as the build runs, you will also have the results conveniently at hand once the Testing step is complete. After the test run, you will see a high-level overview of how many tests failed, passed or were skipped on the Results tab as well as a detailed test log on the Log tab.

Among other details, the test log contains information about the status of the test on each testing device as well as used device minutes.

Run status on device pool 'android pool 1': COMPLETED (sylph run timeout: 410 of 720)
       device: Google Pixel 2 , passed: 2, failed: 1, minutes: 7.62
Run 'sylph run 2019-09-16 06:35:44.741' completed 1 of 1 jobs.
 Result: FAILED
 Device minutes: 7.62 (5.25 metered).
 Counters:
   skipped: 0
   warned: 0
   failed: 1
   stopped: 0
   passed: 2
   errored: 0
   total: 3
Warning: run failed. Continuing...

More detailed information on the test run, including videos, device logs and data about memory and CPU usage, is available in your AWS Management Console.

Wrap-up

Any tests are always better than no tests. While you can run Flutter integration tests on iOS simulator or Android emulator, testing your app on physical devices is a step that really helps to fine-tune its performance and eliminate possible device-specific bugs. Thanks to Sylph, AWS Device Farm is now integrated into the Codemagic build workflow so you can effortlessly verify that your important releases not only pass the CI check but also pass the integration tests on a pool of real Android and iOS devices.

Note that using AWS Device Farm requires an AWS account and you will be billed separately by AWS for device usage. If you don’t have an AWS account yet but would like to benefit from testing on real devices, contact us at info@codemagic.io to discuss an alternative option.

Codemagic CI for Flutter