Mobile Test Strategies

Application development for mobile devices is a fast growing phenomenon.

Users want mobile applications to be simple and fast. Just one nagging bug or usability issue can spoil the entire experience. And with so much competition in this new space, if users don’t have an excellent experience with your application, they will switch to a rival product faster than you can say “There’s an app for that.”

When planning your testing effort for a mobile device application, in addition to the usual functional testing, it is also important to consider the following areas and how they differ from desktop or regular web applications:

1. User Interface Testing 

Mobile devices have unique user interfaces like smaller screens that can be re-oriented, touch screens and soft keyboards, and navigation methods like hard keys and trackballs.

The first area to explore in your test plan is the user interface. While Smartphone applications are relatively new, there are already accepted guidelines for look and feel and overall behavior. It is your job as the tester to confirm that your application.

1. Overall color scheme/theme of the device
2. Style and color of icons
3. Progress indicators when pages are loading
4. Menus. How they are invoked and typical items they contain
5. Overall responsiveness of applications on this device

Touch screens

Multi-touch vs. single touch screens:

If your device and application support multi-touch features, like the pinch-to-zoom effect on iPhone, be sure to include lots of test cases involving touching the screen in more than one place simultaneously, especially while typing on the soft keyboard.

Long touch vs. short touch

While there is usually no concept of a double-click on touch screen devices (although there could be if specifically implemented in your application), some, like the Android smart phones, distinguish between long touches and short touches: pressing and holding an item will bring up a context menu in the middle of the screen, while short-clicking the same item will automatically perform the first action in that context menu.

Button size and position

Ensure that buttons and icons are large enough and far enough from the edges of the screen to be easily clicked by a large fingertip.

Trackballs, track wheels, and touchpad’s

If your device doesn’t have a touch screen, it is even more important to verify that screen navigation is as painless as possible for the user. In these cases, the user may rely on a trackball, track wheel, or touchpad to move from object to object

Soft keyboards

There are often various special cases and corner cases that are important to end users.

1. Does the soft keyboard automatically appear if the user’s main action is to enter some text?

2. Does the first layer of the soft keyboard include shortcut “@” and “.com” keys if the highlighted field is for entering email addresses?

3. Does a long touch on a soft character key bring up several different character choices for that input?

4. Can the soft keyboard be dismissed and re-displayed easily?

5. Can the soft and hard keyboards be used interchangeably (if the device has both)?

6. Do soft keyboard characters entered in password fields only show up as ****?

Hard keys

Be sure to include a lot of testing around the use of the device’s available hard keys such as Start, Home, Menu, and Back. These should all interact with your application similarly to how they interact with the device’s native applications.

2. External Factors Testing
   
   
   Mobile device applications must also contend with interactions and interruptions from other device features like various network connection types, SD cards, phone calls, and assorted device settings.

Network connection Types

App going to be used on devices in various locations with various network connection speeds, it is important to plan testing coverage for the following scenarios:

1. Only Wi-Fi connection

2. Only 3G connection

3. Only 2G connection

4. with no SIM card in the device

5. In Airplane mode (or all connections disabled)

6. Using the network through a USB connection to a PC

7. And most interestingly, test intermittent network scenarios that a user might encounter in the real world:

a. Walk out of Wi-Fi range so the connection automatically switches to 3G/2G (for example, in a Large building like a hospital or airport, or outdoors).

b. Ride in an elevator or on a train where the network connection may go up and down

c. No network connection available at all

SD card interactions

If your application can potentially store or retrieve items on the device’s SD card, then it is important to test the application’s behavior when there is an SD card present or not. At a minimum, the application should provide user-friendly error messages when a function cannot be performed due to a missing SD card.

Phone calls and other interruptions

If device you're testing with is also capable of making and receiving phone calls, be sure to test

The following scenarios,

1. Your application is interrupted by an incoming call, originator hangs up the call

2. Your application is interrupted by an incoming call, terminator hangs up the call

3. Your application is interrupted by placing an outgoing call, originator hangs up the call

4. Your application is interrupted by placing an outgoing call, terminator hangs up the call

Also consideration such interruptions as,

1. Text messages

2. Voicemail notifications

3. Calendar events

4. Social media notifications (Face book, Twitter, etc)

5. Alarm clocks

6. Low battery notifications

Device options

1. Sound profiles

2. Device password/unlock pattern

3. Font

4. Screen timeout/Auto on, off

5. Screen orientation

6. Connections

3. Stress Testing

Mobile device applications have much less overall device memory and power available so must handle them very efficiently.

Stress testing is a must to find exceptions, hangs, and deadlocks that may go unnoticed during functional and user interface testing.

1. Load your application with as much data as possible to try to reach its breaking point
2. Perform the same operations over and over again
3. Perform the repeated operations at varying speeds – very quickly or very slowly
   4.Leave your application running for a long period of time, both interacting with the device and just letting it sit idle, or performing some automatic task that takes a long time.

(E.g. – Slideshow)

5. Randomly send screen taps and keystrokes to your application.

6. Have multiple applications running on your device so you can switch between your application and other device applications often.

4. Security Testing 

Mobile device security will become more and more important as the user base grows, so it is essential to test the security of your mobile web applications, sensitive data storage, and how your application behaves under various device permission schemes.

1. Most mobile devices assume one user; they do not have multiple accounts. (Except in the case of multiple email addresses configured per device, which should be tested especially if your application deals with the device’s Contacts or Email functions). But in general there is no concept of user switching, different profiles, or permissions based on user level.
2. It is up to the user whether or not they configure a password/unlock pattern for their device at all.
3. Outside communication of any sensitive data should be done over SSL/TLS because it can’t be assumed that the user will configure their Wi-Fi to be secure.

4. Emulator Use

Emulators can be a great asset when it comes to achieving testing coverage on multiple devices, but your test plan must also respect the fact that sometimes there is just no substitute for the real thing.

1. Not all activities can be realistically emulated, like switching network connections, or taking a picture or video.
2. Some activities just don’t work at all on emulators, like streaming video on a Blackberry emulator
3. Due to lower device power and memory, your application could exhibit slower performance overall when run on an actual device.
4. If the emulator and actual device have different dpi resolutions, your screens may not display as you expect.

In general, it is a good idea to use some combination of real device and emulator testing,