Flutter web performance testing on Codemagic

Nov 12, 2020

Use M1 Mac mini VMs by default with Codemagic🚀 Build faster

Written by Souvik Biswas

Maintaining consistent performance without jaggedness is an important factor for improving user experience. Performance testing helps developers to pinpoint which component is causing the hiccup and on which page of the web app.

There are two broad types of performance testing:

Quantitative testing looks at metrics like response time.
Qualitative testing is concerned with scalability, stability, and interoperability.

In this article, we will take a look at the quantitative testing part of a Flutter web app and how you can run it in the CI environment of Codemagic.

Sample app

The sample Flutter web app that we will be using is called Coffee Brewery. It contains a list of coffee descriptions and a floating action button that increments the number of coffees in the cart.

Selecting any coffee leads to another page showing that coffee’s description.

Setting up for testing

In order to run Flutter web performance benchmarks, we can use the package called web_benchmarks_framework. It helps in running performance tests on Chrome.

dependencies:
  web_benchmarks_framework:
    git:
      url: https://github.com/material-components/material-components-flutter-experimental.git
      ref: f6ebb4ed3b6489547d9ae58216df9999112be568
      path: web_benchmarks_framework

This package is adapted from macrobenchmarks and devicelab, two packages used by Flutter for web performance testing of the Flutter Gallery.

Also, it is recommended to use the most recent dev channel release of the Dart or Flutter SDK for running these benchmark tests on the web.

To switch to the dev channel of Flutter:

flutter channel dev
flutter upgrade

To switch to the dev channel of Dart (on macOS):

brew unlink dart
brew install --head dart

You can check out this guide for instructions on switching to the dev channel on Windows or Linux.

Adding tests

The things we are going to test are some simple user actions, like:

Scrolling through the list
Navigating between two pages
Tapping the floating action button (FAB)

Create a new folder called benchmarks under the lib directory, and add a new file called runner.dart:

We will be defining the tests in this file. Add the following:

/// A recorder that measures frame building durations.
class AppRecorder extends WidgetRecorder {
  AppRecorder({@required this.benchmarkName}) : super(name: benchmarkName);

  final String benchmarkName;

  bool _completed = false;

  @override
  bool shouldContinue() {
    if (benchmarkName == 'scroll') {
      return profile.shouldContinue();
    } else {
      return profile.shouldContinue() || !_completed;
    }
  }

  @override
  Widget createWidget() {
    final automationFunction = {
      'scroll': automateScrolling,
      'tap': automateTapping,
      'page': automatePaging,
    }[benchmarkName];
    Future.delayed(Duration(milliseconds: 400), automationFunction);
    return MyApp();
  }

  Future<void> automateScrolling() async {
    final scrollable = Scrollable.of(find.byKey(listItemKey).evaluate().single);
    await scrollable.position.animateTo(
      2000,
      curve: Curves.linear,
      duration: Duration(seconds: 5),
    );
  }

  Future<void> automatePaging() async {
    final controller = LiveWidgetController(WidgetsBinding.instance);
    for (int i = 0; i < 10; ++i) {
      print('Testing round $i...');
      await controller.tap(find.byKey(listItemKey));
      await animationStops();
      await controller.tap(find.byKey(backKey));
      await animationStops();
    }
    _completed = true;
  }

  Future<void> automateTapping() async {
    final controller = LiveWidgetController(WidgetsBinding.instance);
    for (int i = 0; i < 20; ++i) {
      print('Testing round $i...');
      await controller.tap(find.byIcon(Icons.add_shopping_cart));
      await animationStops();
    }
    _completed = true;
  }

  Future<void> animationStops() async {
    while (WidgetsBinding.instance.hasScheduledFrame) {
      await Future<void>.delayed(Duration(milliseconds: 200));
    }
  }
}

Future<void> main() async {
  await runBenchmarks(
    {
      'scroll': () => AppRecorder(benchmarkName: 'scroll'),
      'tap': () => AppRecorder(benchmarkName: 'tap'),
      'page': () => AppRecorder(benchmarkName: 'page'),
    },
  );
}

Understanding the tests

The AppRecorder class extends the WidgetRecorder, which helps in recording the performance data as it drives the app.
The shouldContinue method causes the AppRecorder to stop recording frames after all gestures have finished.
The automateScrolling method simulates the action of scrolling through the list of coffee descriptions by finding one element using its key.
The automatePaging method simulates the action of navigating to the description page of a coffee type and then returning back to the home page.
The automateTapping method simulates the action of tapping the floating action button (FAB), which in turn increments the number of coffees present in the cart.
animationStops repeatedly checks whether an animation is happening and stops when all animation has stopped.
runBenchmarks is a function that allows us to select which benchmark to run and displays the results in the browser.

Running the tests

You can run the tests on Chrome by running the following command from the root project directory:

flutter run -d chrome -t lib/benchmarks/runner.dart --profile

This uses runner.dart as the entry point, instead of main.dart.

On running this command, you will see the following screen, which lets you select the test you want to run.

On selecting any test, it will run the benchmark and generate the result. The following is the animation for the “page” benchmark:

After the test completes, it will generate a result like this:

Collecting data from Chrome’s DevTools

You can generate the result of the benchmark in a JSON format. Create a file called run_benchmarks.dart in the test folder, and add the following:

import 'dart:convert' show JsonEncoder;

import 'package:web_benchmarks_framework/server.dart';

Future<void> main() async {
  final taskResult = await runWebBenchmark(
    macrobenchmarksDirectory: '.',
    entryPoint: 'lib/benchmarks/runner.dart',
    useCanvasKit: false,
  );
  print(JsonEncoder.withIndent('  ').convert(taskResult.toJson()));
}

You can run the tests by using the following command:

dart test/run_benchmarks.dart

This will print the result in the console (in JSON format), like this:

Received profile data
{
  "success": true,
  "data": {
    "scroll.html.preroll_frame.average": 50.72727272727273,
    "scroll.html.preroll_frame.outlierAverage": 635,
    "scroll.html.preroll_frame.outlierRatio": 12.517921146953405,
    "scroll.html.preroll_frame.noise": 0.1557479535529603,
    "scroll.html.apply_frame.average": 159.76530612244898,
    "scroll.html.apply_frame.outlierAverage": 1624.5,
    "scroll.html.apply_frame.outlierRatio": 10.168039854378234,
    "scroll.html.apply_frame.noise": 0.4763820247107002,
    "scroll.html.drawFrameDuration.average": 643.1318681318681,
    "scroll.html.drawFrameDuration.outlierAverage": 3167.8888888888887,
    "scroll.html.drawFrameDuration.outlierRatio": 4.925722151027576,
    "scroll.html.drawFrameDuration.noise": 0.26153072675117545,
    "scroll.html.totalUiFrame.average": 2046,
    "tap.html.preroll_frame.average": 37.72222222222222,
    "tap.html.preroll_frame.outlierAverage": 64.2,
    "tap.html.preroll_frame.outlierRatio": 1.701914580265096,
    "tap.html.preroll_frame.noise": 0.14439149898135345,
    "tap.html.apply_frame.average": 298.0777777777778,
    "tap.html.apply_frame.outlierAverage": 1047.1,
    "tap.html.apply_frame.outlierRatio": 3.512841540239311,
    "tap.html.apply_frame.noise": 0.9220792123340406,
    "tap.html.drawFrameDuration.average": 719.6145833333334,
    "tap.html.drawFrameDuration.outlierAverage": 7415.75,
    "tap.html.drawFrameDuration.outlierRatio": 10.305169144362578,
    "tap.html.drawFrameDuration.noise": 0.6977711839517265,
    "tap.html.totalUiFrame.average": 465,
    "page.html.preroll_frame.average": 45.44565217391305,
    "page.html.preroll_frame.outlierAverage": 62.625,
    "page.html.preroll_frame.outlierRatio": 1.3780196125328867,
    "page.html.preroll_frame.noise": 0.227067133294334,
    "page.html.apply_frame.average": 509.4888888888889,
    "page.html.apply_frame.outlierAverage": 1582.9,
    "page.html.apply_frame.outlierRatio": 3.1068390980067173,
    "page.html.apply_frame.noise": 0.732141578034883,
    "page.html.drawFrameDuration.average": 1389.2105263157894,
    "page.html.drawFrameDuration.outlierAverage": 5804,
    "page.html.drawFrameDuration.outlierRatio": 4.177912483424891,
    "page.html.drawFrameDuration.noise": 0.5862849206826566,
    "page.html.totalUiFrame.average": 2153
  },
  "benchmarkScoreKeys": [
    "scroll.html.drawFrameDuration.average",
    "scroll.html.drawFrameDuration.outlierRatio",
    "scroll.html.totalUiFrame.average",
    "tap.html.drawFrameDuration.average",
    "tap.html.drawFrameDuration.outlierRatio",
    "tap.html.totalUiFrame.average",
    "page.html.drawFrameDuration.average",
    "page.html.drawFrameDuration.outlierRatio",
    "page.html.totalUiFrame.average"
  ]
}

If you want to learn more about the results, take a look at this post.

Configuring for Codemagic

You can use codemagic.yaml to run the Flutter Web performance tests on Codemagic and also to collect the benchmark result in a JSON file.

First, you have to make some changes to the test/run_benchmarks.dart file to store the result in a file. Modify it as follows:

import 'dart:convert' show JsonEncoder;
import 'dart:io';

import 'package:web_benchmarks_framework/server.dart';

Future<void> main() async {
  final taskResult = await runWebBenchmark(
    macrobenchmarksDirectory: '.',
    entryPoint: 'lib/benchmarks/runner.dart',
    useCanvasKit: false,
  );

  // For storing the benchmark result to a JSON file
  final result = File('report.json').openWrite(mode: FileMode.write);
  result.write(JsonEncoder.withIndent('  ').convert(taskResult.toJson()));

  await result.close();
}

Now, you can use the following YAML template for running the tests on Codemagic:

workflows:
  flutter-web:
    name: Performance testing
    environment:
      flutter: dev
    scripts:
      - name: Switch to dart dev
        script: |
          brew unlink dart
          brew install --head dart          
      - name: Enable flutter web
        script: flutter config --enable-web
      - name: Get flutter packages
        script: cd . && flutter packages pub get
      - name: Run benchmarks
        script: dart test/run_benchmarks.dart
    artifacts:
      - report.json
      - build/web/
    publishing:
      email:
        recipients:
          - user@example.com # Enter your email here

Understanding the YAML contents

Uses to the dev channel of Flutter.

Switching to the dev channel of Dart:

brew unlink dart
brew install --head dart

Enabling Flutter Web:
```
flutter config --enable-web
```
Getting Flutter packages:
```
flutter packages pub get
```
Running the performance tests:
```
dart test/run_benchmarks.dart
```
Retrieving the benchmark result:
```
artifacts:
  - report.json
```

Running on Codemagic

Add the codemagic.yaml file to the root project directory, and commit it to your cloud repository (GitHub, GitLab, Bitbucket, etc.).

Follow the steps below to start the build:

Go to the Codemagic Applications dashboard.
Search for your project, and go to its settings.
Click on Start new build.
Select the workflow from codemagic.yaml, and click on Start new build.

This will start a new benchmark test with the selected workflow on Codemagic.

Tests successful

Conclusion

Once you have the benchmark results, you can assess if anything is causing performance hiccups and iterate upon improving the performance of your Flutter web app. You can also configure codemagic.yaml to run the benchmarks as new commits are made in order to retain performance as new features are added.

References

Souvik Biswas is a passionate Mobile App Developer (Android and Flutter). He has worked on a number of mobile apps throughout his journey. Loves open source contribution on GitHub. He is currently pursuing a B.Tech degree in Computer Science and Engineering from Indian Institute of Information Technology Kalyani. He also writes Flutter articles on Medium - Flutter Community.

Search for articles

Sorry, but we couldn't find any matches...

Popular articles

Making the Leap from Azure to Codemagic

How Tuist migrated from GitHub Actions to Codemagic for faster and more reliable CI

Codemagic helps Mobi Lab deliver mission-critical projects while saving them time and money