Testing Techniques for Mobile Applications

There are 3 types of mobile applications

1 Browser Based application

• Applications built for mobile browsers
• Can be accessed by entering the specific URL in mobile browser. 
• No Installation/Un-installations involved
• No Upgrade involved
• Browser Based Application users always expect connectivity
  • Network speed and coverage become an important test case 
• Speed is a critical aspect: 
  • User entered values can’t be stored locally in device database, until user allows storing information in Cache, he has to enter all his information manually every time.

2 Pre-installed application

• Applications which are shipped as in built software with the mobile device
  • If devices are not launched yet, a prototype of that device model is given by the device manufacturer. 
• No Downloads involved.
• No Installation/Un-installations involved
• Automatic Upgrades can be done.
  • This is specific to application.
• Testing of these applications in prototypes is very critical.
• They deal with the very core data and functions of the device.
  • If anything goes wrong, the core database might get corrupt.
• They cannot be uninstalled /deleted ever.
• Crashes can cause severe damage to ROM 
• Only way to remove these applications would be to flush ROM and then upgrade it.

3 Installable applications

• Can be upgraded:
  • This is specific to the application design 
  • This also depends on the source from where application was downloaded 
• Applications whose executable files/packages can be downloaded / received by wireless media or wired media. 
  • Platform Specific
• Can be installed/uninstalled in the device.
• They can be downloaded from:
• App Store 
  • OTA (Over The Air) 
  • A URL which leads to download of the application.
    • Transfer via USB cable
  • Can be transferred from wireless media like Blue Tooth, Infra Red etc.

Mobile testing generics

Testing in various network types:

• 2G:
  • GPRS
  • CDMA
  • EDGE
• 3G
• Wi-Fi
• Different types of Plans based on service provider

Monitoring Battery consumption patterns

• Observe battery consumption rate as the application is being
  • Run in Background 
  • Run in Foreground 
• Run the application for long time.

Check memory usage patterns when in device apart from the pre installed applications

• No other applications are installed
  • Lot of free memory available in device.
• Some third party applications are installed
  • Lesser free memory available in device
• Lot of third party applications are installed
  • Very less free memory available in device

Memory Consumption Pattern (RAM usage)

• Check memory consumption pattern when multiple applications are running while the application is being:
  • Launched
  • Run
    • In Background
    • In Foreground
  • Exit

Interruptions:

• Activities which can occur parallel within the device while application is being:
  • Installed
  • Launched
  • Run
  • Exit
  • Upgraded
  • Uninstalled/ Deleted

Examples of Interruptions:

• Incoming call
• Receiving incoming call
• Receiving message
• Device shutdown
• Remove battery
• Camera activated
• Lose network connectivity and then regain it

Debug Build:

• A very important asset in helping reproduce “difficult to reproduce” bugs
• A build released in debug mode which has logs enabled.
• Enabling logs based on sequence of keys
• Logs can be read continuously, and can help retrace steps to any issue, provided application is running.

Build Signing:

• Required for a build which has to be pushed to an app store.
• Confirms that norms as per the authority of OS or device manufacturer are followed.

Testing round after build signing needs to be done

• Ensures the final build being submitted is of appropriate quality standard.

Platforms covered

• iPhone
• Android
• BlackBerry
• Symbian
•  Windows Mobile
• J2ME Applications

Input models in different platforms

• Touch Screen
  • Testing with single touch inputs
  • Testing with multiple touch inputs
  • Testing for Virtual Keypad
• Track Ball/ TrackWheel
• Keypad
• Test cases around multiple inputs at the same time

Device modes in different platforms

iPhone

• Additional Testing aspects:
• UI Guidelines from Apple need to be adhered to.
• Backward OS compatibility
• Allows running only one application at a time.
• Can debug via USB cable connection
• Device Reboot not much needed until for clearing cached data

iPhone testing tool

• Screenshots:
  • Can be taken by clicking the Home key and the Lock key together.
• No other explicit tool needed
• Memory Sweep
  • A tool with which a memory snapshot of iPhone device can be obtained at any given time.

Android

Additional Testing Aspects:

• Allows running multiple applications in background
• Application gets normally minimized on exiting.
  • So a reboot needed to launch application again.
  • Reboot needed often

Android - Debugging

• Select "Enable USB Debugging" in device settings.
• Then connect device to system via USB cable.

Android -Testing Tools

• Dalvik Debug Monitor Server (DDMS)
  • Port-forwarding services
  • Screen capture on the device
  • Thread and heap information on the device
  • Logcat
  • Process, and radio state information
  • Incoming call and SMS spoofing
  • Location data spoofing, and more.
• Monkey Tool
  • Runs on simulator or device
  • Generates pseudo-random streams of user events such as clicks, touches, or gestures, as well as a number of system-level events.
• TaskKiller
  • A utility application used to kill process.
  • Useful to skip device reboot.

Blackberry

Additional Testing Aspects:

• Allows running multiple applications in background
• Device Reboot
  • Not much needed until for clearing cached data
  • Needed mainly for while uninstalling/installing any application.

Build Signing/Verification

• Signed/certified by RIM

BlackBerry – Debugging

• USB cable connected debugging
• Can be done via JDE console after attaching debugger

Testing Tools

• Native Logs:
• A combination on Alt+LGLG keys opens the device logs in device.
• Has 3 types of filters:
  • Debug
  • Information
  • Error
• This helps analyze type of exceptions.

JL_Cmder:

• A little batch script to manipulate JavaLoader.
• JavaLoader, a tool shipped with JDE, can be used to easily:
  • Display Device Info
  • Display an event log
  • Take a screenshot
  • Wipe the handheld
  • Reset it to factory conditions

BBScreenshooter

• Desktop Based tool
• Needs device to be connected to desktop system for capturing screenshots

CaptureIt

• Device Based tool
• Device need not be connected to desktop for capturing screenshots.

Symbian

Additional Testing Aspects:

• Allows running multiple applications in background
• Device Reboot not much needed until for clearing cached data

Build Signing/Verification

• Signed/certified by Symbian
• By device manufacturing authority like Nokia

Symbian - Debug

• Can be connected to the IDE via:
  • USB cable connected debugging
    • More preferred
  • Via BlueTooth
    • Less Preferred due to device driver issues

Testing Tools

• Screenshot tools:
  • All tools are device based
  • After installing the tool
    • Set a shortcut key for screen capture.
    • Start the tool and then let it run in the background while application is run.
    • Clicking on the shortcut key captures screenshot and stores it in device.
  • Examples:
    • ScreenSnap S60 (5th Edition)
    • ScreenTaker for Nokia 7650/3650/N-Gage/SX1
• Yucca Tools:
  • File browsing in Symbian device
  • Crash Monitor
  • Shows all supported fonts in device with preview.
  • Memory status

Windows mobile

Additional Testing Aspects

• Allows running multiple applications in background
• Can exit the application from task manager
  • Needed in case application gets hanged in background and its process needs to be killed

Windows Mobile

• Debugging
• Debugging
  • USB Cable connection needed between device and desktop
  • Can use visual studio

Testing Tools

• Hopper
  • For stress testing.
  • Throws random events such as click events, entering text if text field in application run in background etc.
  • Logs are stored in device; screenshot is also stored if device hangs or if application crashes.

J2ME applications

J2ME is a Development Platform

Input Modes:

• Follows input modes of the platform, some restrictions might exist.

Device Modes:

• Based on platform

Additional Testing Aspects:

• For mass markets handsets
• Porting Friendly
  • Should work well on the supported device/ platform on which its installed.
  • Should be able to work well after the device firmware upgrades.

Build Signing/Verification

• Verified by Verifying Authority.
• Verified by device manufacturing authority like Nokia

Testing Tools

• Tools provided by device platforms can be used.
• Wireless Toolkit
• Debug mode build.

Testing on simulator

Simulators:

• A software application that can accurately imitate mobile phone.

The need to test on simulators:

• Non availability of device
• Testing same build in multiple device screens
• Non availability of certain network for device testing in testing area.

Advantages:

• Can help in isolating issues which are not volatile network connection dependent.
• Can provide a wide variety of testing over different types of device simulators for the same build.

Disadvantages:

• Simulators of older generation handsets don't resemble the device as closely
• Some issues which are hit by the speed at which input was given cannot be reproduced easily
• Hardware/Firmware environment variations detectable in device testing only
• Device testing is always preferred as it represents more likely end user scenarios