With mobile Internet usage now representing more than 50% of total Internet traffic, providing a positive mobile experience for your customers is a top priority. The good news is that testing for a great user experience is easier than ever, providing your team has the knowledge and tools.
When I started in quality assurance (QA) back in 1997 at NEC, we had to test software on physical devices, signing out various peripherals from a lab, and manually installing the software onto laptops. When testing mobile apps today, smaller companies, startups and those with budget limitations may have to take a similar approach. Larger companies, and those with significant exposure and risk, should consider the rich feature set of a cloud-based mobile testing solution. Prices for cloud solutions vary widely, as do their capabilities. Though you will pay enterprise-level pricing for an enterprise-level mobile cloud testing solution, the investment is worth it. Your marketing and product teams need to be included in the decision making also, as this is a solution that can set the entire company up for success, not just "another tool for the QA team."
Before jumping to a cloud solution, there are a couple of areas that are sometimes overlooked that should be examined. Prior to starting development, your team needs to have requirements for mobile application analytics. Imagine releasing an application and having it crash 2% of your users, but you don't know why -- all you do know is that your app just got another 1-star review in the Apple store. Adding an analytics package from the start will ensure problems reported post release can be corrected quickly. It can also show you what features are used, and which ones are not, so you can focus on delivering value with each release. Next, you also need to review the requirements for the Apple Store and Google Play Store to ensure your application will not be rejected when you submit it. If your app is rejected, you might need to add 4 more sprints worth of unplanned work to fix what should have been built right the first time.
Ideally, development teams conduct formal code reviews, write programmatic unit tests, and execute security scans and static code analysis. These best practices help prevent the introduction of defects, therefore reducing the cost and time of future rework. Despite being a best practice, these activities still represent opportunities for improvement in many companies. I find it best to embed the QA team into these practices to ensure these activities are really getting done. Once these steps have been completed, you are ready for functional testing.
Functional Testing: The Road Ahead
When looking at the number of mobile device combinations in the market, there may be thousands of combinations, and testing all of them is not practical. Testing only four is too little, testing 700 combinations is too many. If you can test on 40 to 60 combinations of Android and iOS phones and tablets, you will have great coverage. Maintaining 40+ physical phones in your test lab requires a lot of non-value-added work. Time is wasted cleaning and restoring the images, devices get lost, locked in desks when people are on vacation, or get damaged in various ways. Fortunately, cloud solutions can simplify the management of these devices while providing additional features.
A cloud-based testing solution allows you access to several devices remotely. Though some cloud-based testing solutions use emulators, there are cloud vendors that have actual mobile devices in the cloud. And while features vary by vendor, you can typically get access to all the common Apple, Android, Surface, and other devices along with all the supported operating system versions and betas.
Cloud solutions also have reporting, device allocation and user/group access. These features allow you to allocate devices among your team, and allow remote users to use the devices without having to run into the office to "get some phones to test with." Some companies still are reticent about moving anything to the cloud. A mobile test solution might be a good way for them to build up trust in the cloud. You will need to work with your infrastructure, firewall and security teams to create a VPN from your office to your cloud provider, but nothing here is anything they haven't done already.
Automated testing is a necessity if you want to run many tests on more than a handful of devices. You should connect your testing framework to your cloud device solution. Testing frameworks such as xFramium allow you to do this. Your QA team should be able to write one set of test scripts that work on all devices, assuming your Android and iOS programs are in functional sync, meaning one isn't four months ahead of the other. It's best to ensure that after the development team creates a build, the build deploys and installs to your test cloud devices, runs all your tests each build across every device in your cloud, and then gives you a report. This DevOps approach is supported by many mobile cloud vendors.
Aside from the specified functionality of your application, there are many other issues to consider. These include switching between Wi-Fi and mobile networks on the fly, Bluetooth, camera, audio, GPS antenna virtualization, device memory, device heat monitoring, system log review, program and device interruptions, performance testing, security testing and future betas for new phones and operating systems.
Though all the content for mobile application development and delivery could fill a book, the above will get you off to a great start. If your company is developing its own mobile applications, a cloud testing solution should be considered. If you are outsourcing the entire effort, you may wish to include many of the points mentioned above in your SLAs to ensure a successful deployment to your clients.