Releasing a new mobile app or new features to existing apps is always stressful. No matter how much testing you have performed before deploying apps to production, there is always some suspense until you see the app running in the production without any bugs. There are multiple levels of testing to be performed before deploying a new feature to production. The following are the main types of testing performed during the development of the app.
- Unit Testing (Developers) to test an unit of code.
- Integration Testing (Developers or Technical QA) to test different units as an integration.
- Network/Service testing (Developers or Technical QA) to test API contracts or network layers.
- UI Testing (Developers or Technical QA) to test user journeys of apps as a real user would.
- Manual Testing/Alpha Testing (Non-Technical QA or anyone in the team) to test the apps as a real user without writing any codes.
These kinds of tests are always represented in the form of a pyramid which is also known as the testing pyramid.
Reference: https://martinfowler.com/articles/practical-test-pyramid.html
You can learn about the testing pyramid here. The definition of the testing pyramid changes in every testing conference depending on who the speaker is, but the pyramid shown above is the basis of all of the testing stages. As you can see, there is a lot of testing going on in the development process of mobile apps. Not every team can perform all kinds of testing, but the majority of the teams following modern application development practices have most of the testing practices in place.
So, the question is: what is beta testing and how do companies do beta testing for mobile apps?
Beta Testing
Beta testing is the process of distributing the app for testing to the big audience before launch. The goal of beta testing is to fix bugs and get user feedback before the launch of the app to real users. In the process of beta testing, a group of the general users uses the version of your app before launch. This is a non-public and limited release of your app.
Alpha Testing vs Beta Testing
The alpha testing phase is performed just before beta testing with people who have actually been involved in the development of the app or people within the same team, department or company. Alpha testing is carried out using either black-box (without knowing code) or white-box (knowing code) methodologies. In the alpha testing phase, testers are already aware of the product or feature going to production. Alpha testing can be done in-house and in the form form of acceptance testing. Alpha testing is usually carried out at the end of the product development, when the app is almost ready for release, and before the beta testing starts. The goal of alpha testing is to address bugs and get the product working for the beta testing phase.
On the other side, beta testing is performed before distributing the product to the customers. Beta testing is only performed in a real production-like environment with a limited set of real users who might not be involved in product development. The goal of beta testing is to address the issues and feature requests from real users before releasing the app to the customers. There is an awesome guide on Alpha testing and Beta testing here if you want to know more.
Why Beta Testing?
As we have seen, an application goes through an extreme testing process before coming to the beta testing phase. If the application development team has performed all sorts of testing before beta, then what’s the need for the beta testing in real life? Can an internal QA team do these kinds of tests? The answer is NO.
Well, if that were the case, then there couldn’t be any bugs in an app which has 100% unit test coverage or solid automated testing in place. In real life, configuring the production-like environment for testing is hard and expensive in the early stages of development. There are a lot of scenarios that can not be replicated in the in-house environment. Testing performed by tools isn’t always guaranteed to work in the production environment. It’s difficult to effectively test the app by creating virtual users or configuring an environment which acts like a real environment.
How Does Mobile Beta Testing Work?
Most of the companies make the mistake of skipping the beta testing phase and releasing the app directly to the production. This mistake can cost a lot if the critical bug is transferred to the production. Beta testing can avoid this situation if the beta test is conducted smartly. In mobile application development, especially iOS and Android, we have to plan beta testing as soon as acceptance testing is finished. There are various things needed to be considered while defining the strategy of beta testing to get the most out of the beta testing phase. Some of the checkpoints are:
- Find out where to get beta testers.
- How many beta testers are required for the launch? What kinds of testers are you looking for, e.g technical or non-technical, or test to break attitude?
- Define the onboarding process for the beta testers, so that they can get going straight away without any issues.
- Give them an easy-to-download option without any technical details.
- Make sure a crash reporting tool is set up before the beta testing starts.
- Make the feedback or bug reporting template available for them, so that issues or feedback reporting processes can become faster.
- Consider the beta testers' feedback and act on it immediately.
- Keep the beta testers engaged with the app and support them during the testing phase.
This is a small checklist, but you can plan the beta test effectively as per the requirements and budget of the app. Remember that the beta testing phase does not only give you fixed bugs but great ideas and feature enhancements too, so it’s worth considering.
Available Mobile Beta Testing Platforms
There are several beta testing platforms available on the market for iOS and Android. There are also official beta testing platforms available for iOS and Android apps which are provided by Apple and Google.
Official Beta Testing Platforms
TestFlight (iOS)
TestFlight is Apple’s official solution for the beta testing of the iOS and other Apple platform apps. When an iOS build is uploaded to App Store Connect, we can easily get it using the TestFlight App. In order to get the builds on TestFlight, you must enter with your Apple Developer account and have to be a member of the development team. When the build is submitted for review, it’s possible to add external testers or share a public link to anyone to get the latest build on the devices.
Google Play Console (Android)
Google Play Console is Android’s official beta testing platform provided by Google. You can learn more about Google Play Console here.
Third-Party Beta Testing Platforms
There are so many third-party solutions available on the market, but we will mention only some of them over here.
- Fabric/Crashlytics
- HockeyApp (Visual Studio App Center)
- TestFairy
- Testmagic by Nevercode
Take a look at the advantages and disadvantages of these beta testing platforms.
Why Not to Just Go with Official Platforms?
You might be wondering why companies need to look for a third-party solution for beta testing when there is official beta testing support provided by Apple and Google. It’s true that official solutions are great, but they have some platform-specific limitations.
- TestFlight requires an Apple Developer account and only supports iOS and Apple platform apps. TestFlight apps can only be available to external testers when the app is submitted for review.
- Google Play Console doesn’t support iOS and the user needs to have a Google account.
- Official solutions are hard to integrate with other tools used internally in the companies.
There might also be other limitations, so companies go for the third-party solution to meet the demand for both iOS and Android apps.
Features of a Great Beta Testing Platform
Now we have seen some of the beta testing platforms available on the market. So, what makes a beta testing tool a perfect solution for mobile app development? Here are some of the features you should look for in a beta testing tool:
- Support for both iOS and Android;
- Easy Over The Air distribution;
- Ability to add release notes to the new features;
- Team and group management, so that a particular build can be sent to certain people;
- Devices/Build/Report Management;
- Crash Reporting;
- Smooth user interface to provide feedback or report a bug;
- Integration with bug tracking systems, e.g JIRA or BugZilla.
This is a small list, but there might be other features that you might be looking for in your mobile project.
Conclusion
Beta testing is an important phase of mobile application development to reduce risk while deploying new apps or features. Selecting good mobile beta testing tools can help teams to plan their beta testing strategy and get the most out of beta testing. In 2019, you should keep an eye on new beta testing tools coming on the market which can fulfill your project requirements.
