Mobile application testing has always been challenging due to all the variants in mobile devices and OS platforms.  Mobile testing has gotten more efficient over the years as new mobile development platforms and middleware aim to reduce application complexity and standardize functionality into a “one size fits all” application.  In addition, as mobile emulators are now developed and made readily available to the application development community, testing accuracy has greatly improved.  Furthermore, with the introduction of new automated testing tools and hardware testing approaches, physical hardware testing can be accomplished without even the need for device procurement and account provisioning.

Having said that, there are still a few areas where testing can be challenging.  In my experience, some real world user experience testing scenarios that are still challenging to test are:

• Application behavior during intermittent connectivity lost, i.e. walking through a department store while executing a transaction.
• Application server timeout, i.e. a hung server process or an overloaded server that is slow to respond.
• Roaming partner gateway configuration and granular billing analysis, i.e. configuring APN gateways and validating roaming charges, and transaction line item matching associated to roaming charges
• Application performance while on roaming partner network, i.e. segregating roaming network hoping time from application performance time
• SMS and USSD short code testing through partner carrier networks globally, i.e. ensuring short codes are valid and working in all geographic carrier networks.

Most of the test scenarios listed above still require some combination of manual testing or configuration of system resources and coordination of several cross-functional team resources to emulate the exception paths.

Posted on: Monday, October 26, 2009