Enterprise Mobile Application Testing in Cloud – How critical is it?

We have seen remarkable growth in the mobile technology industry in the past few years.  There have been so many advancements in terms of both hardware and software that continuously push the software developers to utilize these capabilities and build newer and more exciting business interfaces. The “app stores” by Google, Apple, Windows, and Amazon have set the strategy in mobile software development and guided the path to new areas of revenue.

This growth spurt has increased market competition so much that application vendors are pressured to develop apps with a short turnaround time in order to survive in the market. Testing these apps in a mobile environment is where the real challenge lies. After all, you wouldn’t want a brilliant idea wasted just because it wasn’t tested enough right?

The statistics that the “CapGemini World Quality Report 2013-14 Mobile Testing Pull Out” recently revealed is actually quite mind-boggling. While 65% organizations reported that they do not have the right tools to test, the problem for 52% of them was that they did not have the right devices readily available to test out their applications. According to the report, 34% mentioned the lack of right testing processes/methods as the reason for their shortcomings whereas 29% companies out in the market accepted that they did not have an expert app evaluator. An even more surprising conclusion from this report was that 19% companies don’t have a quality in-house testing environment while another 18% of these organizations shockingly did have enough time to test out their applications.

Will it suffice to just test the functional correctness of an enterprise application? Of course not.  There are so many other components of a successful application and this includes performance, security, usability, vulnerability, OS compatibility, hardware compatibility etc. Though there is lot to write, in this article I am going to highlight only about enterprise mobile application functionality testing and how the testing cloud can help in minimizing the challenges mentioned above.

Some of the most challenging tasks in testing enterprise mobile applications:

• Rapidly changing device hardware
• Diversity of devices, platforms and OS versions
• Varying screen sizes and form-factors
• Network capabilities (2G, 3G, 4G, wireless, etc.)
• No guarantee of continuous network connectivity (Sometimes no network)
• Use of enterprise data in public network
• Minimum keys to use with ability to touch interface and gestures
• Limited testing tools in market and finding the right tool to test with

To overcome these challenges, the testing strategy we adopt is very important and has to be planned well in advance. Using emulators only serves half the purpose and is complete only when tested on an actual device.

In short, a winning strategy has to cover a large sample set covering a multitude of devices. This can be achieved by strategically establishing the real devices to test across the testing lab, emulators and actual devices from the cloud. Just testing the application functionality alone cannot be a successful plan because it is as equally important to include a plan for testing connection stabilities for various geographic locations. Also how is the app going to behave when you decide to make a call midway or send a text or get notifications from other applications?

Though there is an extensive permutation of devices, platforms and OS versions possible, it is almost virtually impossible to test the app on every device. What do you do then? How do you identify the targeted devices? The answer to this seemingly difficult question is actually simple. You just request the marketing team to research the device that has great penetration in a specific demography and model your application testing around that device.

Network stability testing is also a part of functionality testing. Some additional testing like lower end device hardware, buffering capability, memory leak, battery consumption etc., is also very critical before releasing the app because it is tightly tied up with the application functionality. Lets look at some of the best practices in the industry and what tools can best help us adopt these practices.

Some of the best testing practices that can help you create a successful application include

• Setting up a testing strategy and plan
• Building a broad spectrum of test cases to cover any scenario possible
• Identifying best suited automated testing tools
• Completing device emulator testing and make it functionality-error free (Testing coverage is 20% to 30%)
• Testing in standard physical devices that are part of the in-house mobile testing lab (Testing coverage is 95% to 100%)
• Testing in a secure private testing cloud (Testing coverage is 85% to 95%)

Automated testing tools
The three key principles that form the foundation to a successful and proper automation methodology include re-usability, maintainability, and expandability.

There isn’t a one suit fits all methodology here. The automation process itself has to be customized for every organization that best suits their culture and process.

Device emulator testing

Here’s the most exciting part of using emulators – this can be done with absolutely zero investment. Isn’t that great? The trick here is, the application is tested without using any real physical device. Another advantage of emulators is that, in addition to functionality testing, they can be used to test stress, smoke, performance and compliance conformation.

Testing in in-house labs

In-house labs give a more realistic view of test results and function best when equipped with the devices that are popular with a specific demography. On a quarterly basis, existing models should be evaluated and updated to meet current requirements as needed.

Testing in cloud

The “In Cloud” testing methodology is an on-demand service that is fast growing in popularity wherein, the physical devices are accessed through web interface or client applications that are pre-installed on the tester’s desktop. Any mobile application testing cloud supports deploying, managing and testing the application in the device cloud. Some clouds also support demography specific calls, texting and push notification interfaces.

The next question – how exactly do you use the cloud? The process is actually as clear as water

• Connect to testing cloud using web or native client
• Identify and select the device(s) to test
• Upload and install the app on all selected devices
• Execute test
• View the device screen in desktop (Real-time)
• Download the test log
• Uninstall the app from the device in cloud and clear the cache
• Release the device back to cloud

With so many cloud systems available in the market right now, how do you identify a good testing cloud? In my opinion, a cloud can only be efficient if it supports

• Web based testing and files management
• Installation, cleaning and removing the app directly in the cloud device
• Unattended environment monitoring to plan and schedule overnight regression testing
• Both public and private device cloud
• Device blocking capability for later testing with SLA on device availability
• Multitude of operating systems like iOS, Android, BlackBerry, Windows, etc.
• Manual and automated testing
• Examining vital device data like CPU, memory, battery usage etc.
• Retrieval of device logs with support to screenshot
• Actual device components like camera, OCR etc.
• Live sharing of test sessions using videos and screenshots
• Integration with multiple automation tools
• Performance testing, network testing and monitoring
• Connecting to 3^rd party APIs

Here is the link to compare between two popular mobile application testing clouds, Perfecto and DeviceAnywhere.

Conclusion

Before setting the mobile application testing strategy, the strategist has to be clear with the business expectations, user demography, user community, solution landscape with key business areas and targeted mobile devices. The real success lies in identifying the proper testing tools, automation methodology, choices of physical devices in lab and last but not least, the perfect mobile application testing cloud. To conquer the competition and minimize cost, a structured and organized enterprise mobile application testing strategy is very critical.

Happy Testing!!!!

Cheers

Venkat Alagarsamy