The beta testing phase is key to achieving the successful release of mobile apps. In our previous post, we have covered the ultimate guide for mobile beta testing.
In this post, we will introduce the main mobile app testing platforms available on the market and cover the advantages and disadvantages of these vendors.
Mobile Beta Testing
Beta testing is the phase where you distribute apps to a bigger audience before launch. The beta testing environment is considered as a production-like environment where users can find the real issues. No matter how much testing effort we have put into other kinds of testing like unit testing, integration testing, functional testing, or manual or exploratory testing, the beta testing stage shouldn’t be avoided. Beta testing gives the confidence that your app still works when distributed to a larger audience.
In the process of beta testing, you need to have a platform from where you can distribute the apps to the larger audience or external testers. There are various platforms available on the market to support the beta testing of mobile apps. Some platforms are from Apple or Google itself, while others come from third-party companies. It’s always challenging to select the right beta testing platform considering your project need and budget.
Mobile Beta Testing Platforms
Most of the businesses develop apps for both Android and iOS platforms, so it’s essential to have a beta testing platform which supports the need for getting both iOS and Android apps without jumping to other beta testing platforms. Official beta testing platforms provided by Apple and Google have this limitation that beta testers need to be switched from one platform to another. Therefore, companies are looking for beta testing platforms which can easily be used for both iOS and Android apps. Now let’s explore what the options available on the market are for the beta testing of mobile apps.
Official Solutions
TestFlight (iOS)
TestFlight is Apple’s official solution for the beta testing of iOS and other Apple platform apps. Apple acquired TestFlight in 2014 to support the beta testing needs of the Apple platform apps. When Apple platform apps are uploaded to the App Store Connect, we can easily get it to use the TestFlight App. In order to get the builds on TestFlight, you must have an Apple Developer account and have to be a member of the development team. When the build is submitted for review, we can add external testers or share a public link to anyone to get the latest build on the devices. App Store Connect has limited the number of internal testers but you can get up to 10,000 external testers. However, your app needs to be submitted to Apple’s review process in order to get the external testers. Testflight allows us to create different groups and assign different builds to different groups. It supports testing 100 different apps at the same time and the builds can be sent to the users via a public link. You can watch a quick intro video of TestFlight by Apple here.
The Advantages of TestFlight
- TestFlight is an official tool managed by Apple so you will get a production-like environment.
- TestFlight makes the distribution process easy as the apps can be sent to beta. testing using public links or adding the email addresses of the external testers.
- TestFlight is free of charge.
- TestFlight has the possibility for testers to add notes, an area to provide feedback about the app.
- Better control on how many testers should be involved in the beta testing process.
- Notification when a new version of the app is uploaded to TestFlight.
The Disadvantages of TestFlight
- TestFlight only supports Apple platform apps.
- The app needs to be submitted for review before sending it out to the beta testers.
- TestFlight doesn’t have a monitoring dashboard to monitor crashes and the number of active users.
Google Play Console (Android)
Traditional terms like ‘alpha testing’ and ‘beta testing’ have been abandoned by the Google Play Console in favour of ‘internal’, ‘closed’ and ‘open testing’. The Google Play Console is Android’s official beta testing platform provided by Google. You can learn more about the Google Play Console here. Google has three different kinds of tests but the open test is closely related to beta testing. The apps set for the open test can be visible in the Play Store and can be downloaded by anyone. There are no limits to how many apps you test or how many testers you wish to involve in the process. You can read the detailed process of Google’s beta testing approach here.
The Advantages of the Google Play Console
- An official solution by Google.
- No limits in the numbers of apps and testers.
- Choose the test types, e.g internal, closed or open.
- Open test apps are visible on the play store and can be downloaded from there.
- Control the number of testers for the open tests.
The Disadvantages of the Google Play Console
- Only supports Android or Google platform apps.
- Before beta testing, the app needs to go through Google’s approval process.
- Multi-build testing isn’t supported in the open test.
Third-Party Solutions
There are hundreds of third-party solutions available on the market for the beta testing of mobile apps but we will cover only the popular ones. The most common beta testing services used by companies are:
- Crashlytics Beta (Fabric);
- HockyApp/App Center;
- TestFairy;
- DelployGate.
We will cover only these given tools in brief.
Crashlytics Beta (Fabrics)
Crashlytics Beta (Fabric) was started on Twitter and then the tool was acquired by Google in 2017. Crashlytics and Fabric is an open-source SDK that can be integrated into any app. However, Google is trying to integrate it with Firebase. Crashlytics combined with Fabric can be used for both iOS and Android platforms. It reports crashes but also offers the distribution of apps and app analytics services. With Fabric, you can distribute the apps to the wider group. The beta testers will get an email invite to download the Crashlytics Beta app and get a version of the app for testing. There is no limit to how many testers you can invite to test the build but if you are testing the ad-hoc builds, there is a limitation of 100 devices per device type (iPhone, iPad) by Apple. If you have an enterprise distribution certificate, then you don’t have that limit. Crashlytics integrates with bug tracking software like JIRA, so that you can create the issue easily.
The Advantages of Crashlytics Beta
- Fabric SDK is free.
- Ability to create groups of testers.
- Crash reporting and app analytics features.
- Works for both iOS and Android.
The Disadvantages of Crashlytics Beta
- Limits on the numbers of devices.
- Distributing apps requires scripting and embedding SDK in the apps.
- Testers cannot provide feedback directly.
HockeyApp (Visual Studio App Center)
At the moment HockeyApp is in the transition of fully becoming a part of Visual Studio App Center. HockeyApp used to be a popular tool for beta testing and app distribution.
Microsoft HockeyApp is one of the most popular beta distribution services on the market. Many large companies are attracted towards HockeyApp due to its unique and amazing features. It is well suited for large companies that develop many different apps. The reason why Microsoft named it HockeyApp is quite interesting. It supports many platforms like Android, iOS, macOS and Windows phone, also React Native, Xamarin and Unity. Apple calls installing beta apps on iOS devices “ad hoc”. The name Hockey was a wordplay on the ‘Hoc’ in ‘ad hoc distribution’ and the word ‘key’. When they launched the HockeyApp the open-source project had already become popular in the community, so Microsoft kept the same name and added App at the end.
As HockeyApp is being decommissioned, there is a plan by Microsoft to make all the HockeyApp features available in the App Center. Similarly to Crashlytics it provides crash reporting, tester management and notification features. Beta testers can get the link to download the apps and the apps can be installed just by visiting the URL.
The Advantages of HockeyApp
- Managed by a big corporation like Microsoft.
- Crash reporting and app analytics.
- Works on both iOS and Android platforms.
- Manage the groups and testers from the Dashboard.
The Disadvantages of HockeyApp
- HockeyApp features will be controlled by Microsoft after the transition into App Center.
- Some of the existing features will change after the transition.
TestFairy
TestFairy is another popular mobile beta testing platform. It can be used for beta testing native apps or cross-platform apps developed using tools like React Native or Xamarin. TestFairy SDK needs to be integrated into the app in order to enable features like crash reporting. One of the key differences between TestFairy and other platforms is that it offers a video recording feature while testing the app. It becomes easier to report an issue. TestFairy targets enterprise users and has great API to integrate with third-party tools. Unfortunately, TestFairy is not free.
The Advantages of TestFairy
- Advanced features like video recording and mobile insights.
- Can be hosted using a private cloud.
- App Feedback and easy integration with other tools.
The Disadvantages of TestFairy
- Not free.
- No pricing plans not clear.
Conclusion
Mobile beta testing is an essential stage while releasing apps to customers. There are hundreds of beta testing tools available on the market at the moment, but selecting the right tool can be challenging. It’s a great idea to keep an eye on the market and see what the current trends in mobile beta testing are.
