Friday 26 April 2013

Five-Things to keep in mind before starting Mobile Application Testing



Before starting testing any mobile applications (let it be .chatting tools, social networking, games, business apps) there are some things that a Mobile Application Tester should go through for effective testing.

1. Analyzing similar applications:- Try to analyze some other application which are similar to your application. For example if you have to test any media sharing application on Mobile just search for some other media sharing applications and observe its feature.

2. Keep your emulator ready for testing : Some times it takes times for processing any request for example for downloading any media files or for loading an page on device. In this case to save time you may try some test with your emulator so that this time will be utilized and overall time in testing will be reduced.

3. Analyze the device related issues:- When it is deviced which are the target devices do not forget to have a look on device related known issues. This will help you understand which are the issues related to device and which are due to your application under test.

4. Use emulator but don’t completely trust it:- While testing you may take help of emulator but please note that all the test cannot be performed in emulator. Also in emulator response time is faster, so it may happen you may miss some issue which comes in weak network on actual devices.

5. Define the performance criteria:-For any mobile applications performance is one of the most important concern. Make sure you are having some performance parameters so that you will be testing the mobile applications against it. Since Memory is one of the constraints for mobile devices performance and behavior of your application under these conditions is interesting things to see.

Tips for Mobile-Web Testing


1. You can use Firefox user-agent Add-on that supports mutitple Os and devices
2. Use Firefox add-on to check only UI issues, Functional things you have to verify on devices for better result, because functionally is not per selected browser, it will consider Firefox itself
3. Use Firesizer Firefox add-on with user-agent addon, using that u can change screen resolution
4. For localization quick check - use any pc browser, you can easily set the language here. Android/BB device may/may not have the language required.
5. Performance tools use 3pmobile - android/BB keynote MITE - Utilize trail version many websites are available where we can paste our url and get the results for example- w3c checker
6. Main things to check

  • Whether screen fits in the display.
  • Navigation options , Hyperlinks and  image display.
  • Scroll options.
  • Website URL should be short and easy to type.
  • There should be a shortcut to Home page
  • Browser Cookies/Cache checking.
  • Font and Size
  • Control size displayed
  • Tappable control should not be too small
  • Overall UI
  • Receive call while downloading files check whether the download is getting disturbed.
  • Check the speed of the browser.
  • Performance by downloading large memory files and video streaming.
  • Check whether u can navigate forward n backward.
  • Add bookmarks.
  • Disable show image option.

7. If you find difficult to type URL in android emulator/phone
quick option is - adb command adb shell am start -a android.intent.action.VIEW -d http://www.stackoverflow.com

Mobile Test Strategies


Application development for mobile devices is a fast growing phenomenon.
Users want mobile applications to be simple and fast. Just one nagging bug or usability issue can spoil the entire experience. And with so much competition in this new space, if users don’t have an excellent experience with your application, they will switch to a rival product faster than you can say “There’s an app for that.”
When planning your testing effort for a mobile device application, in addition to the usual functional testing, it is also important to consider the following areas and how they differ from desktop or regular web applications:
  1. User Interface Testing 
Mobile devices have unique user interfaces like smaller screens that can be re-oriented, touch screens and soft keyboards, and navigation methods like hard keys and trackballs.

The first area to explore in your test plan is the user interface. While Smartphone applications are relatively new, there are already accepted guidelines for look and feel and overall behavior. It is your job as the tester to confirm that your application.

  1. Overall color scheme/theme of the device
  2. Style and color of icons
  3. Progress indicators when pages are loading
  4. Menus. How they are invoked and typical items they contain
  5. Overall responsiveness of applications on this device

Touch screens

Multi-touch vs. single touch screens:
If your device and application support multi-touch features, like the pinch-to-zoom effect on iPhone, be sure to include lots of test cases involving touching the screen in more than one place simultaneously, especially while typing on the soft keyboard.

Long touch vs. short touch
While there is usually no concept of a double-click on touch screen devices (although there could be if specifically implemented in your application), some, like the Android smart phones, distinguish between long touches and short touches: pressing and holding an item will bring up a context menu in the middle of the screen, while short-clicking the same item will automatically perform the first action in that context menu.

Button size and position
Ensure that buttons and icons are large enough and far enough from the edges of the screen to be easily clicked by a large fingertip.

Trackballs, track wheels, and touchpad’s
If your device doesn’t have a touch screen, it is even more important to verify that screen navigation is as painless as possible for the user. In these cases, the user may rely on a trackball, track wheel, or touchpad to move from object to object

Soft keyboards
There are often various special cases and corner cases that are important to end users.

1. Does the soft keyboard automatically appear if the user’s main action is to enter some text?
2. Does the first layer of the soft keyboard include shortcut “@” and “.com” keys if the highlighted field is for entering email addresses?
3. Does a long touch on a soft character key bring up several different character choices for that input?
4. Can the soft keyboard be dismissed and re-displayed easily?
5. Can the soft and hard keyboards be used interchangeably (if the device has both)?
6. Do soft keyboard characters entered in password fields only show up as ****?

Hard keys
Be sure to include a lot of testing around the use of the device’s available hard keys such as Start, Home, Menu, and Back. These should all interact with your application similarly to how they interact with the device’s native applications.

  1. External Factors Testing

    Mobile device applications must also contend with interactions and interruptions from other device features like various network connection types, SD cards, phone calls, and assorted device settings.

Network connection Types
App going to be used on devices in various locations with various network connection speeds, it is important to plan testing coverage for the following scenarios:
1. Only Wi-Fi connection
2. Only 3G connection
3. Only 2G connection
4. with no SIM card in the device
5. In Airplane mode (or all connections disabled)
6. Using the network through a USB connection to a PC
7. And most interestingly, test intermittent network scenarios that a user might encounter in the real world:
a. Walk out of Wi-Fi range so the connection automatically switches to 3G/2G (for example, in a Large building like a hospital or airport, or outdoors).
b. Ride in an elevator or on a train where the network connection may go up and down
c. No network connection available at all

SD card interactions
If your application can potentially store or retrieve items on the device’s SD card, then it is important to test the application’s behavior when there is an SD card present or not. At a minimum, the application should provide user-friendly error messages when a function cannot be performed due to a missing SD card.

Phone calls and other interruptions
If device you're testing with is also capable of making and receiving phone calls, be sure to test
The following scenarios,
1. Your application is interrupted by an incoming call, originator hangs up the call
2. Your application is interrupted by an incoming call, terminator hangs up the call
3. Your application is interrupted by placing an outgoing call, originator hangs up the call
4. Your application is interrupted by placing an outgoing call, terminator hangs up the call

Also consideration such interruptions as,
1. Text messages
2. Voicemail notifications
3. Calendar events
4. Social media notifications (Face book, Twitter, etc)
5. Alarm clocks
6. Low battery notifications

Device options
1. Sound profiles
2. Device password/unlock pattern
3. Font
4. Screen timeout/Auto on, off
5. Screen orientation
6. Connections

  1. Stress Testing
Mobile device applications have much less overall device memory and power available so must handle them very efficiently.
Stress testing is a must to find exceptions, hangs, and deadlocks that may go unnoticed during functional and user interface testing.
    1. Load your application with as much data as possible to try to reach its breaking point
    2. Perform the same operations over and over again
    3. Perform the repeated operations at varying speeds – very quickly or very slowly
      4.Leave your application running for a long period of time, both interacting with the device and just letting it sit idle, or performing some automatic task that takes a long time.
(E.g. – Slideshow)
5. Randomly send screen taps and keystrokes to your application.
6. Have multiple applications running on your device so you can switch between your application and other device applications often.

  1. Security Testing 
Mobile device security will become more and more important as the user base grows, so it is essential to test the security of your mobile web applications, sensitive data storage, and how your application behaves under various device permission schemes.
    1. Most mobile devices assume one user; they do not have multiple accounts. (Except in the case of multiple email addresses configured per device, which should be tested especially if your application deals with the device’s Contacts or Email functions). But in general there is no concept of user switching, different profiles, or permissions based on user level.
    2. It is up to the user whether or not they configure a password/unlock pattern for their device at all.
    3. Outside communication of any sensitive data should be done over SSL/TLS because it can’t be assumed that the user will configure their Wi-Fi to be secure.


  1. Emulator Use
Emulators can be a great asset when it comes to achieving testing coverage on multiple devices, but your test plan must also respect the fact that sometimes there is just no substitute for the real thing.
    1. Not all activities can be realistically emulated, like switching network connections, or taking a picture or video.
    2. Some activities just don’t work at all on emulators, like streaming video on a Blackberry emulator
    3. Due to lower device power and memory, your application could exhibit slower performance overall when run on an actual device.
    4. If the emulator and actual device have different dpi resolutions, your screens may not display as you expect.

In general, it is a good idea to use some combination of real device and emulator testing,


QA Checklist Mobile App Testing

  1. Introduction


This checklist is specifically designed to test the characteristics of a mobile app. Obviously, it tests only generic app characteristics and not the functionality of the app. For this a separate test approach and test script must be created. The same goes of course for performance testing, usability testing, security testing and other testing activities necessary for your specific app.
The checklist is split into five different fields:

  • Device specific characteristics. These are characteristics that are related to the device on which the app is installed.
  • Network specific checks
  • App checks. These are things to check that have to do with functionality that is frequently used in an app.
  • App User interface checks.
  • Store specific checks.

  1. Device specific checks

#
Description
OK/
NOK?
Can the app be installed on the device?
N/A


Does the app behave as designed/desired if there is an incoming call?
N/A


Does the app behave as designed/desired if there is an incoming SMS?
N/A


  • Does the app behave as designed/desired if the charger is connected?
N/A


  • Does the app behave as designed/desired if the charger is disconnected?
N/A


  • Does the app behave as designed/desired if the device goes to sleeping mode
N/A


  • Does the app behave as designed/desired if the device resumes from sleeping mode
N/A


  • Does the app behave as designed/desired if the device resumes from lock screen?
N/A


  • Does the app behave as designed/desired if the device is tilted?
N/A


  • Does the app behave as designed/desired if the device is shaken?
N/A


  • Does the app behave as designed/desired if a local message is coming from another app (think of: calendar reminders, to-do task etc.).
N/A


  • Does the app behave as designed/desired if a push message is coming from another app (think of: twitter mentions, whatsapp message, wordfeud invitation, etc).
N/A


Does the app interact with the GPS sensor correctly (switch on/off, retrieve GPS data)?
N/A


Is the functionality of all the buttons or keys on the device defined for this app?
N/A


Verify that buttons or keys which have no defined function have no unexpected behaviour on the app when activating.
N/A


In case there’s a true “back” button available on the device does the “back” button take the user to the previous screen?
N/A


In case there’s a true “menu” button available on the device, does the menu button show the app’s menu?
N/A


In case there’s a true “home” button available on the device, does the home button get the user back to the home screen of the device?
N/A


In case there’s a true “search” button available on the device, does this get the user to some form of search within the app?
N/A


Does the app behave as designed/desired if the “Battery low” message is pushed
N/A


Does the app behave as designed/desired if the sound on the device is turned off?
N/A


Does the app behave as designed/desired if the device is in airplane mode?
N/A


Can the app be de-installed from the device?
N/A


Does the application function as expected after re-installation?
N/A


Can the app be found in the app store? (Check after go-live)
N/A


Can the app switch to different apps on the device through multitasking as designed/desired?
N/A


Are all touch screen positions (buttons) working when a screen protector is used.
N/A


  1. Network specific Checks


#
Description
OK/
NOK

Does the app behave according to specification if connected to the internet through Wi-Fi?
N/A
Does the app behave according to specification if connected to the internet through 3G?
N/A
Does the app behave according to specification if connected to the internet through 2G?
N/A
  • Does the app behave according to specification of the app is out of network reach?
N/A
  • Does the app resume working when it gets back into network reach from outside reach of the network?
N/A
  • Update transactions are processed correctly after re-establishing connection.
N/A
  • Does the app still work correctly when tethering or otherwise connected to another device
N/A
  • What happens if the app switches between networks (Wi-Fi, 3G, 2G)
N/A
  • Does the app use standard network ports (Mail: 25, 143, 465, 993 or 995 HTTP: 80 or 443 SFTP: 22) to connect to remote services, as some providers block certain ports.
N/A



  1. App specific Checks


    #
    Description
    OK/
    NOK?
    Has the app been tested on different type of devices and different versions of OS?
    N/A
    Stability check: if the app has a list (for instance of pictures) in it, try scrolling through it at high speed.
    N/A
    Stability check: if the app has a list (for instance of pictures) in it, try scrolling to before the first picture or behind the last picture.
    N/A
    Is downloading of the app prevented in case it’s bigger than the OS allows downloading when connected to cellular networks.
    N/A
    • Integration: does the app connect correctly to the different social networks (LinkedIn, twitter, facebook, etc).
    N/A
    • The app does not interfere with other apps when in background/multitasking mode (using GPS, playing music, etc.).
    N/A
    • Can the user print from the app (if applicable)
    N/A
    • The search option in the app displays relevant results
    N/A
    • Verify most common gestures used to control the app.
    N/A
    • What happens if you select different options at the same time (undesired multitouch, for example – select two contacts from the phone book at the same time).
    N/A
    • App name should be self explanatory
    N/A
    • Does the app limit or clean the amount of cached data.
    N/A
    • Reloading of data from remote service has been properly designed to prevent performance issues at server-side. (manual reloading of data can reduce the amount of server calls)
    N/A
    • Does the app go to sleep mode when running in the background (prevent battery drain)
    N/A
  2.  App User interface checks

This checklist is based on the recommendations of Apple and some other experts. This checklist is not a substitute for a usability test, to get a good feeling of the user experience of an app a user experience test is always the most reliable method. User interface checks toelichten.

#
Description
OK/
NOK?
    1
To keep controls as unobtrusive as possible for instance by fading them out if they are not used for a while.
N/A
          2
Make it possible for users to go back to a previous screen for instance by adding a back or cancel button
N/A
  • The main function of the app should be apparent immediately. It should speak for itself.
N/A
  • Use at most one action on the screen that is highlighted as the most likely for the user. (Example: in iOS a blue button represents the default or most likely action).
N/A
  • Minimize user actions by using a picker or a table view where users can select a certain choice over a data entry field where users have to type a choice
N/A
  • In an app, the user should not be able to store files locally, outside the app sandbox.
N/A
  • In an app, the user should not be exposed to the permissions of a specific file
N/A
  • If there is a long list of data to scroll trough, provide a search option above the list.
N/A
  • If performance is slow, indicate a progress status icon (“Loading…”), preferably with specific message.
N/A
  • In case of ‘live’ filtering of data while the user enters his search query, verify the performance.
N/A
  • The appearance of buttons that perform standard actions are not altered in the app (for instance: refresh, organize, trash, Reply, back, etc.)
N/A
  • Do not use standard buttons for other functions then that they are normally used for
N/A
    3
The app should respond to all changes in device orientation, as per the design
N/A
  • Tapable elements should be about 7x7 mm in size, using the pixel density of the target device you can calculate the amount of pixels (chapter documentation contains a link to different devices compared).
N/A
  • Do not redefine gestures in your app that have a standard meaning (example: swiping from top to bottom enables the notification center)
N/A
  • Requirement to login is delayed in the app as long as possible
N/A
  • If the app is stopped at an unexpected time, user data should be saved locally and available at start-up.
N/A
  • Users should be warned of the consequences of deleting a document
N/A
  • Keyboard adjusts to expected input (for instance numbers/letters when expected).
N/A
  • Are inactive buttons clearly distinguished from active buttons?
N/A


  1. Store specific Checks

These checks are important to be confident that an Apple app will pass the submission process of the app store.

These checks are largely based on the apple App store guidelines. The other leading app store, Google Play, has much less strict demands. Other app stores like the Amazon app store have stronger restrictions, we’ve added them in a special paragraph in case they are relevant.

Keep in mind that the Apple app store submission process is not crystal clear. On many occasions, it is unclear why apps are rejected or accepted. Keep the following line in mind from the apple app review guidelines:

We will reject Apps for any content or behaviour that we believe is over the line. What line, you ask?
Well, as a Supreme Court Justice once said, "I'll know it when I see it". And we think that you will also know it when you cross it.

Note: the guidelines that are covered here are the ones that are special to the app store. For instance: Guideline 2.1 states that “Apps that crash will be rejected”. Making sure that an app doesn’t crash is not remedied by a simple check – it requires a functional test.

Second Note: Part of the iOS guidelines is the following guideline 10.1 Apps must comply with all the terms and conditions explained in the Apple iOS Human Interface guidelines. We’ve incorporated these in the Chapter “App user interface checks”.

So if you’re not planning to submit the app to an app store, these checks are not necessary? In principle, yes. However, many of these have relevance even if your app is not distributed through a store but for instance to a limited set of your employees. So these checks might still be useful.

#
Description
OK/
NOK?
The app can’t use any “non-public API’s”. This means that you can’t use some functions that the distributing platform uses for its own apps. (This can generally be checked best by some sort of automated tool, like http://www.chimpstudios.com/appscanner/)
N/A
The app can’t reprogram controls of the device that are not intended for that use. (For instance: using the volume button as a shutter for the camera).
N/A
The app should not access information on the device outside the app without the user’s permission (for instance, copying the address book or getting information from other apps).
N/A
The app should not access or write files outside the “Bundle” and “Documents” directory. (because the app can’t read or write data outside the designated container area).
N/A
The app cannot download code to be installed without the users consent.
N/A
The app can only get new functionality by way of an upgrade through the app store.
N/A
After download, an app should remain working. An app cannot turn off after a few days.
N/A
An app can’t be a “trail”, “beta”, “demo” or “test” version.
N/A
Apple product names should be spelled correctly in the app. (For instance: IPhonez is wrong).
N/A
If the app uses the web, it is not done using third party (i.e. non-Apple) browsers.
N/A
You cannot mention other app platforms in your app (for instance: “Also available on android!”)
N/A
An app cannot use old interfaces, like for instance the iPod click wheel.
N/A
Multitasking functionality of the app can only be used for its intended purposes, i.e. VoIP, Audio playback, location, task completion, local notifications, etc. This means that generally an app can’t run in the background but has to be closed off if it’s not used any more.
N/A
The app must have some functionality. For instance, it can’t be just a title page leading to some text. It can’t be just a song, movie or book as there are different platforms for that.
N/A
Functionality should be in sync with functionality described in store.
N/A
In general, the app has to be decent. So no explicit material in the sense of sex, violence, drugs, alcohol or tobacco. It cannot address a specific ethnic or religious group in a derogatory way.
N/A
The app has to be honest. This means that the description of the app has to be correct, and all functionality has to work as described. If an app gives diagnostic information, it has to be reliable. This also means that the genre and category in the description must be appropriate. The app icons should be consistent and appropriate.
N/A
An app can’t restrict the users of the app for instance by location or carrier.
N/A
An app cannot send spam or introduce viruses, or use other apple platforms like Game Center and Push Notifications to do so.
N/A
The app should aim at backing up a minimum of information on iCloud. The information in iCloud should be just the user generated information. Information that can be recreated or downloaded should not be backed up.
N/A
An app cannot use location services of the device without asking permission.
N/A
All url’s in the app code should be fully functional
N/A
The app can’t use the user’s location without permission.
N/A
The location services cannot be used to autonomously control of vehicles or for emergency services.
N/A
An app cannot use push notifications without user consent.
N/A
Push notifications have to be send using the Apple Push Notification (APN) API. This has to be done using an APN ID.
N/A
Push notification can’t send personal information.
N/A
The App may not distribute any private information of users (like Player ID) through the game center.
N/A
Ad banners must be hidden when there are no ads available.
N/A
The app must respect copyright of apple and other parties.
N/A
The in app purchase mechanism cannot be used to purchase goods and services used outside the app.
N/A
The in app purchase mechanism cannot be used to collect money for charities. This has to be done through SMS.
N/A
The in app purchase mechanism cannot be used to buy a raffle or lottery ticket directly from the app.
N/A
Apps that encourage the users to use the device in a way that may damage the device will be rejected
N/A
An app cannot require user’s personal information (for instance email address) in order for it to function.
N/A

Thursday 25 April 2013

Principles of Good Bug Reporting


Reporting a good bug is not hard, but many times we do it hurriedly and without putting our full attention into it. As in the case above, the results of a badly written bug are 3-fold:
1. You waste your time reporting a bug incorrectly and most certainly will need to re-write it.
2. You waste the time of the other people reviewing the bug and trying to make sense of what you wrote.
3. You harm your good name as a QA Professional, by writing a bug that is either correct but incomprehensible or altogether wrong.
What are the components of a Good Bug?
1. Short and precise title
Each bug should have its individual title. The title gives us the opportunity to quickly understand and remember what the bug is all about.
The title cannot be generic (e.g. Error on Application), or too long. It should be one sentence, providing the highlights of the issue and the way to understand how a user could run into it.
Examples of good and bad titles are (taken from actual bugs!): “Unable to log into the system when username contains foreign characters” and not “User cannot work in Chinese”; or “Log rotation doesn’t work, grows until the disc runs out of space” and not “Server crashes due to log size”.
2. Concise but complete description: Steps to reproduce, consequences of the bug, and suggested behavior
The description of the bug is where the tester can express his “artistic sense” and write what he thinks will help stakeholders to understand and handle the bug.
The most important things to include in the bug are:
i. Steps to reproduce – with all the steps, sometimes even the trivial ones, just to make sure people who need to judge the bug and are not experts in the system can understand it correctly. Make sure to include also the data required to reproduce it (e.g account names, strings, etc).
ii. Consequences of the bug – what will happen to the user if/when she runs into it.
Here again, sometimes it is trivial but other times it isn’t and you need to make sure this information reaches the person who is reading your report.
iii. Suggested/Expected behavior – especially when you are reporting a bug that doesn’t create data-loss or other catastrophes, but on the other hand makes the application uncomfortable or unusable to the end-user.
The description is the place where you can add all the information that you cannot write on any other place, but on the other hand you cannot write to much into it (and expect people to read it) or write irrelevant information.
As a rule of thumb, 3 to 10 lines should be enough in 80% of the cases.
3. Good attachments
Specially true when you need to show GUI malfunctions, error messages, and/or log files that compliment the description of your bug.
One thing to take into account is to provide ONLY the relevant information in your attachment. This means that if you have a long log file attach only what you know is relevant and not all of it, if you have a screen capture cut down the area that is relevant and not all the desktop if it is unnecessary, etc
Use also zip files and compressed formats in order to save space.
4. Complete definition of the categorizing fields
If you work with a
Structured Bug tracking system you will have fields that categorize your issues. Examples of these fields are Module, Infrastructure, Browser, etc.
This fields will help your team to categorize and in some cases reproduce the bug correctly. The only reasons not to fill this fields laziness or carelessness.
5. Correct Severity & Priority Another big Tester Sin is to over prioritize your bugs. Obviously you are happy you found a bug and you want it fixed, but giving it the wrong severity will only make the person in charge of the bug angry and it will lower your chances of having this bug fixed.
You will also automatically harm your name with this person and from now on she will be skeptic whenever you report a critical issue (remember “the boy who cried wolf”…)
If your problem is that you are not sure what severity to give your bug you can either consult with a fellow tester or developer, or ask you lead for a table describing the cases and standards used in your company for this purpose.
6. Follow-up and comment.
You should continue following up on your bugs, and providing comments when necessary. This is specially true when a developer or other stakeholder does not understand the bug, and either rejects it or delays it.
You are not the owner of your bugs, but you certainly have say on them and you should make sure that say is heard.
Depending on your company this last point may not apply, but I think in most companies it does.