How Do You Deal With Real World Testing of Mobile Apps – Part 1

By | July 20, 2015

This is the first entry in a two part article written by Parimala Hariprasad.  The following post, which discusses User Testing, can be read here.

A few years ago, if a product solved an unmet need or a critical task at hand, users were more than happy to use it. Today, users are looking for more: engaging and delightful experiences every day. A better experience is at the top of the user expectations list. According to research, most customers will think less of an organization if their experience is not great. Even more importantly, studies have also found that customers will never trust the organization again if their experience is not great.

Mobile devices, as we know, are mobile and are not designed to be hung on the wall! Without sensors, location services, camera and communication features, we would just have a really lame mini-computer in our pockets. These reasons outline the need to test real devices, under real world conditions and not within the comforts of an office cubicle.

Real world testing


Real World Testing

The mobile industry is going through an ocean of changes. With increasing complexity in hardware and software choices, it’s harder to get good testing done on an infinite number of platforms, devices, test configurations and a wide variety of mobile personas. Given this challenge, it’s not enough to test a mobile app while sitting at a workstation. Network switching happens all the time in the real world, and mobile devices that support Cellular and Wi-Fi can either automatically or manually switch between available networks at any point in time. How a mobile app behaves when a device switches between networks is a key deciding factor with respect to whether users will install and use the app or abandon it within a few minutes. Testing apps in different countries, on different devices, working on different platforms where hardware infrastructure is designed differently is a necessity today.

This is where techniques like field testing and user testing light the path ahead.

The Need for Real World Testing

Consider a music streaming app like Spotify that streams music on the go. Testing this app amidst the luxuries of a cubicle with the best possible infrastructure is not enough. In some locations, users may not have access to a specific network, while others may have dead spots where the connection is cut off completely and a few others may have multiple 2G and 3G towers within a small distance, expecting mobile devices and apps to handle network switching between towers on their own. Users might have completely different experiences as those listed.

Consider a few scenarios in which users might use the Spotify app:

  • A month long road trip while listening to songs on Spotify
  • An overnight car or bus journey where a traveller is listening to music from a saved playlist
  • App usage in specific places like malls that have dead spots (note the high turnout of people in malls over weekends)
  • Streaming music over Wi-Fi about 50m away from the building that hosts the Wi-Fi infrastructure
  • Switching from a 3G tower to a 4G tower and vice versa over a distance of several kilometers
  • Streaming music in weaker network areas where only 2G is available, for example, in tier 2 and tier 3 cities and towns
  • And so on…

Apps like these need testing that moves beyond the four glass walls of a workplace. Such apps need to be tested using different network types, speeds and service providers.

This brings us to field testing mobile apps.

Field Testing

Field testing involves deploying testers in the field to use apps under real world conditions where different networks and service providers are at play. While using mobile apps, testers on the field use or access the app under different network conditions:

Real world testing


  • No network
  • Some Network
    • Cellular
    • Wi-Fi
    • 2G
    • 3G
    • 4G
    • LTE
  • Network Switching
  • Shared network conditions
  • Dead Spots
  • Hot Spots
  • And so on….

Testing in these scenarios can allow stakeholders to be confident enough about the quality of the app to release it into the market.

While testing in the field, testers must use mobile personas and user scenarios to make their tests as realistic and useful as possible.

Mobile Personas

A mobile persona is a fictitious identity that represents one of the user groups who will use the mobile app. Understanding the target user base helps testers come up with ideas to test similarly to how users might use the app.

Testers can think of different contexts and test apps depending on the user group. Using mobile personas, testers may not be able to get into the shoes of the users – simply because the users’ shoe sizes are too different from theirs. However, understanding the personas will help testers be aware of how users might use the app and will help design test ideas pretty close to them.

User Scenarios

A user scenario is a narrative that describes how a particular user persona uses the app and how the app fits into their day to day lives. Identifying scenarios which are specific to each user in a user group is a good starting point for testing. User scenarios can vary depending on mobile personas. For example, a teenager looking for a medical insurance plan on a mobile app’s actions are very different from a 65 year old retiree who is looking for an optimized plan that will help him take care of medical expenses in his old age with a small income. Thinking of these two users helps testers think of different contexts in which they might use the mobile app differently.

Testing on the field in different network conditions is a great way to unearth app problems that are hard to find, sitting at a cubicle. This is where field testing is of profound value in finding several problems that exist only under real world conditions.

The next entry in this two part article will touch upon user testing and different methods that help in identifiable critical pain points faced by users.


About Parimala Hariprasad

Parimala Hariprasad

Parimala Hariprasad spent her youth studying people and philosophy, and in the workplace she applied her learnings to helping create skilled testers. Parimala has been a tester for more than twelve years in domains of CRM, security, e-commerce, and healthcare. Her expertise lies in test coaching, test delivery excellence, and creating great teams that ultimately fire her because they become self-sufficient. Parimala has experienced the transition from web to mobile and emphasizes the need for design thinking. She frequently rants on her blog Curious Tester, tweets as @CuriousTester, and is on LinkedIn. Parimala currently serves as a Senior UX Architect at Amadeus Software Labs India Pvt. Ltd