Testmunk joined Snapchat. For testing services, check out TestObject.

Articles


LifeLine Response Reduces Manual Testing Time Ten-Fold

Posted by on December 10th, 2015

LifeLine Response is a personal protection app that calls 911 when you trigger an alarm. LifeLine Response uses the GPS information on your smartphone to provide emergency responders your exact location and personal identifying information in order to provide you the fastest and most accurate assistance possible.

lifeline-response-screens

When Giacomo Listi, CIO of LifeLine Response, got in touch with testmunk, their life-saving application was being tested purely manually. This meant that the organization kept several devices on hand and manually tested key changes on each device in turn.

lifeline-response-giacomo

Mr. Giacomo Listi, CIO

LifeLine Response faced the following core issues:

  • Manual testing on multiple devices took an exceedingly longer time to accomplish – often several days
  • Release dates would be postponed because of unexpected bugs and new regressions
  • Lack of resources to perform adequate manual testing
  • Lack of a formal testing process
  • A small but growing number of user complaints due to bugs

The Testcase Overview

The first undertaking for test automation of any new app through testmunk is to set up a testcase overview. Lifeline was no exception, and we proceeded to determine what needed to be tested overall as part of a given release. In order to get this baseline in place, we arranged a 45 min conference call with Giacomo to walk us through some of the testing procedures that he and his team had recently gone through.

In particular, we focused on the following questions:

  • What are the top 10 most important use cases for LifeLine?
  • Where have you seen predominantly seen regressions / bugs in the past?

Prior to the session, one of testmunk’s automation engineers had obtained the app and installed it on his phone. This allowed us to follow along as Giacomo walked us through the app. Based on Giacomo’s walkthrough, our engineers experience with the app itself, and several questions, we took copious notes, which were later were transformed into the testcase plan below.

lifeline-response-testcase-plan

Snapshot of some of the testcases

Within less than 24 hours Giacomo received this testcase overview that listed the key use cases identified in our meeting. The testcase plan was designed for dual functionality, acting as a status document, indicating which testcases were ready for immediate automation by marking them as “Done.”

The results were as follows:

Android iOS
Can be automated 74% 76%
Maybe/need more info 8% 6%
Cannot be automated 18% 18%

This armed Giacomo with a list of first automated testcases that could be run within but a few days already, and allowed him to focus his manual testers on those items he and his team still had to execute manually.

Luckily this number was low; only 24-26% of testcases still needed to be executed manually. Another 6-8 % could potentially be tested automatically in future iterations with a bit more work.

Sample Testcases

A strong sample use case unique (to our knowledge) to Lifeline Response is a “thumb mode,” which triggers an automatic call to emergency services, alerting the police if your thumb is pressed to the icon on screen for at least 3 seconds.

In order to automate such a test case, testmunk had to be able to define a minimum length for a long press of at least 3 seconds on a given view. This did not prove to be very difficult, highlighting the strengths of our Calabash framework.

We used the following implementation…

And I press the screen for 3 seconds

and the step definition looked like this:

When(/^I press the screen for (\d+) seconds$/) do |seconds| send_uia_command({:command => 
%Q[target.touchAndHold({x: 300, y: 300}, #{seconds})]}) end

Mission accomplished. Not only were we able to automate the test case, Giacomo could see the results for himself via screenshots from our testmunk dashboard.

lifeline-response-testmunk-dashboard

Screenshot from testmunk’s dashboard

While testmunk’s automation platform offered significant and immediate wins, there were a few testcases that simply could not be automated. Examples of these included:

  • Checking a verification code in their native Message screen
  • Testing the sound quality of spoken messages

The undertaking of scripting over 140 testcases (70 per platform) was executed by two of testmunk’s automation engineers working together. For implementations of this size, testmunk often provides two engineers, in order to get results faster. This allows them to work in tandem, discussing requirements, reviewing pull requests, and to chip in with tips from their individual experiences.

Within less than 4 weeks, LifeLine Response had achieved the following testcase coverage improvements:

Before With testmunk
Testing Time 7-10 hours 55 minutes

Lifeline’s testcases were run via the testmunk device cloud, which covers a large set of real Android and iOS devices. For their regular testing runs, Lifeline decided that executing their automated tests on three different iOS devices and 5 Android devices met their needs.

Of particular note is that during the scripting phase, testmunk received numerous new app versions with new or changing features. For many platforms, this can be a big problem, forcing scripted test cases to be re-written. However, using the page object approach meant this was not a problem for us. A few scripted tests that were impacted by the change could quickly be adjusted. Not only does testmunk’s automation approach prevent a great deal of rework, it allowed us to reuse much of the same test steps (tweaked slightly for OS) across both Android and iOS.

Through a combination of automation expertise, platform features, code framework, large device cloud, and scripting approach, Lifeline was able to cut their testing time to nearly a tenth of what it had been.

About the authors:
Yegor Silyutin and Lukasz Fryc are automation engineers at Testmunk and help some of the biggest iOS and Android apps to establish large scale test suites. Both of them have been working in software/QA engineering for several years and are passionate about tackling mobile automation challenges. Besides their work Yegor loves photography and Lukas is racing with his motorbike.

Testmunk automates mobile app testing

LEARN MORE

Read next


Happy Holidays!

Posted by Robert Bayer on December 21st, 2015

Continuous Integration for Mobile Apps

Posted by Martin Poschenrieder, Lukas Fryc on December 3rd, 2015

Leave a Comment

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>