Perfect Cross-device Testing Plan

When it comes to testing, especially that kind which requires many devices, it is essential to determine appropriate manhours and the importance of service features, before selecting the right devices and testing requirements. In this short article, I am going to introduce some tips on choosing the right devices for your test.

・What is the key element of software testing and quality assurance for multi-device testing?

The bottom line is, to cover the test patterns efficiently.
Then, what should be done in order to achieve the coverage that we want? It is only impossible when we pay attention to the following.

Probability = Market Share

Basically, the more popular the device/model is, the more chances your users will see the errors when those defects are device unique. Therefore, I recommend you take the time to first investigate the market share of relevant OS and device models and made decisions based on the results. However, please keep in mind that it is not yet safe just because you carry out the tests using devices with larger market share and that is the difficult part of multi-device testing.

About (1) ~ (3)

(1) Market share %
An OS/device/model/manufacturer with a large user base.
Make sure to pick up information published by Apple/Google, Google analytics/ App analytics, research forums, sales ranking, etc. so that the higher ranked ones can be covered.

(2) Support areas that cannot be supplemented by market share %
There are always support areas different OS and manufacturers require. Make sure these are taken into consideration and covered.

(3) Special cases
Although the above (1) and (2) are focused on the OS and manufacturers, we will also need to consider behaviors that may occur only with specific device/model.

・Specific examples

OS perspective
For example, in the case of apps that support iOS 10 to 13, there are many variations in consideration of the minor OS version and revision. Since it is not realistic to cover all of them, it is necessary to pick up and select. We would advise giving priority to the order below.
A1: Latest minor versions of OS with high market share
A2: Latest minor version of each major version
A3: Initial minor version of each major version

A3 is necessary in order to discover OS-related defects that occur only in the initial version. Very often, such defects are fixed in the subsequent minor versions, so it is efficient to test the initial version and the latest version of each major version within the relevant support areas as shown in red in the figure below.

By the way, for Android, it is uncommon to have as many minor versions as iOS. However, for some models such as Samsung’s OneUI, some have firmware with a significantly revised UI, so ideally we work with devices that would give coverage per manufacturer.

Perspectives

 Screen resolution – as a perspective

Depending on the resolution of the screen installed on the device, a device-specific problem may occur.
If the resolution is high, it may be displayed as smaller than expected. Contrary, if the resolution is low, it may be displayed larger than expected where the content and characters may be cut off. Even with the iPhone as an example, the resolution varies quite a bit depending on the model. For this reason, when selecting a group of devices, it is desirable to be able to cover all combinations of different screen sizes and resolutions.

 Device performance – as a perspective
You must be aware of the amplitude between the good and bad as in high-end and low-end devices.
The performance I am talking about here is the processing speed dependent on the “CPU / GPU chips and memory size”. Sometimes high-end devices perform at a speed we did not expect them to process, while with low-end devices may not process things within the expected time, and errors such as timeout, stuttering, and abnormal heat may happen.

Conclusion

There are still many other points I would like to note, but I think you can already see that there is a lot to multi-device testing and it is never easy, as it takes good preparation with supporting data points.

Any QA team should know that it could ruin the reputation of an entire product or app, because of very few bad reviews coming from errors specific to minor devices. Each project has its own resources and budget, it is important to find the optimal solution between perspectives of cost and service quality, in order to actively incorporate multi-device testing into the service release process.

SHIFT ASIA offers a multi-device testing service that checks both quality and quantity.
We have solid experience in multi-device testing for various customer verticals. Having a good number of devices, we provide full-cycle support from device selection, device preparation, test requirements preparation, and test execution from a professional perspective.
If you ever encounter problems around device selection to reduce device unique malfunctions, please do not hesitate to consult with us, we are here to help!