Testmunk joined Snapchat. For testing services, check out TestObject.

Articles


How QuizUp Mastered Android Device Fragmentation

Posted by on November 19th, 2014

At the bottom of this article you see the actual testcases that you can use for your own project for free.

In the final days before the QuizUp Android launch, the small team of mobile developers and QA specialists scrambled to work out all of the kinks in their app. They had already launched on iOS, but Android was proving to be a unique challenge. Between them, they had about 10 different Android devices. It seemed that every time they fixed an issue on one device it caused a new issue on another.

Guðfinnur “Guffi” Sveinsson, QuizUp’s QA lead, knew that if they were running into so many issues with this small sample of devices, Android fragmentation would be an even bigger problem on a larger scale. portrait of Guðfinnur Sveinsson

“We needed more devices and more time to test,” but investing in more Android devices was something they didn’t want to do, and spending more time testing on those devices would ultimately delay their launch. At that point, Guffi went in search of an alternative option: “we were looking for something that was fast and affordable.”

There are services that use human testers, but they were too expensive and would take too long to set up. That’s when Plain Vanilla Games, the company behind QuizUp, decided on automated testing. Guffi chose testmunk, a company that let them test on 30 different devices through a remote service with easy to script testcases.

Automated testing wasn’t just affordable, it was fast: “If we had to do this manually, we would be looking at an hour to two hours of testing on each device. Even if we had 30 devices, we didn’t have 30 – 60 hours to spend testing every build.” With automated testing, they were able to run all of their tests simultaneously on all devices in about 30 minutes in total. With the Android launch date staring down at them, that kind of time saving was critical.

The automated testing uncovered five key issues:

Issue 1: Display Inconsistencies

Since Android devices have varying screen sizes and resolutions, getting everything to format properly on the screen was a problem. Plain Vanilla Games found that the early versions of the QuizUp app lacked proper alignment and labels on the smaller screens (Figure 1).

QuizUp alignment issues screenshot

QuizUp alignment issues screenshot

Figure 1 – Display problems on the QuizUp login screen

This caused usability issues, making it impossible for users to login on some devices. In some cases, some labels, such as “OR” and “Already signed up? Login now!” were completely blocked.

Issue 2: The Overlapping Keyboard

The next issue discovered was an issue well-known to frustrate app users, on Android phones with 4.0 and 4.3 inch screen sizes, such as the S3 mini, the S4 mini, the Samsung Nexus 4 S, and the Galaxy S Advance, the keyboard overlapped elements in the birthday wrapper. The result was that users couldn’t tap the “Ok” or “Cancel” buttons without hitting the back button to remove the keyboard (Figure 2).
QuizUp alignment issues screenshot

Figure 2 – Keyboard overlaps the “Ok” widget

While issues like this don’t directly impact functionality, they are annoying to users and could result in the user abandoning the app and/or giving it a bad rating.

Issue 3: Birth Date not Getting Populated

The next issue was much more serious. Some data was not saving correctly, resulting in the birth date not populating (Figure 3). This was a problem across all devices running Android OS 4.0.4, but it hadn’t been discovered during manual testing. Figure 8 shows the testmunk dashboard displaying the test runs on a range of devices, with display errors being highlighted.QuizUp alignment issues screenshot

Figure 3 – Data not being saved on Android 4.0.4 – Screenshot from testmunk.com dashboard

Issue 4: Yes/No Buttons in the Tablet

Testmunk provides screenshots of each step during testing. This revealed GUI inconsistencies that affected usability. For instance, on tablets, the “Yes” and “No” buttons were unreadable (Figure 4), due to using fonts that were too small. While this did not impede function, it could have made the app difficult to use for some people. Accessibility on mobile devices is an issue that’s received a lot of attention recently, and testmunk’s screenshots helped Plain Vanilla Games approach this problem more efficiently.

QuizUp alignment issues screenshot

Figure 4 – Labels are too small on tablets

Issue 5: Error Messages Didn’t Appear

During the sign up process, certain data is required from the user, such as a display name and birthday. If the user failed to provide correct data they should have received an error message indicating the problem. Screenshots revealed that these error messages were not being displayed (Figure 5). This made it difficult for users to create new accounts.

QuizUp alignment issues screenshot

QuizUp alignment issues screenshot

Figure 5 – Comparison of screenshots before (left) and after (right) the error message problem was fixed

Automated testing made it possible for Plain Vanilla Games to test their QuizUp app dozens of times before the scheduled launch. A process that could have taken weeks to complete using manual testing was completed within a short time period. Not only was the process fast, it was thorough; problems were discovered that had been overlooked during previous manual testing. The final result is a clean app that delivers a consistent experience to users regardless of device or OS.

QuizUp Calabash tests to check UI inconsistencies

Follow the guide on docs.testmunk.com to use some of the following teststeps for your own project for free.

Feature: Quizup

Scenario: Login screen
  Given I wait for "Login now!" to appear
  Given I wait for 3 seconds
  Then take picture
  Then I press view with id "textView4"
  Then I wait for the view with id "email_input" to appear
  Then I wait for 2 seconds
  Then take picture

Scenario: Signup touch menu
  Given I sign up
  Then I press view with id "ivDialogCloseButton"
  Then I wait for 2 seconds
  Then take picture
  Then I press view with id "top_bar_btn"
  Then I wait for 2 seconds
  Then take picture

Scenario: Signup touch menu -> topics
  Given I sign up
  Then I press view with id "ivDialogCloseButton"
  Then I wait for 2 seconds
  Then take picture
  Then I press view with id "top_bar_btn"
  Then I wait for 2 seconds
  Then take picture
  Then I press "Topics"
  Then I wait for "Arts" to appear
  Then I wait for 2 seconds
  Then take picture

Scenario: Signup touch menu -> profile
  Given I sign up
  Then I press view with id "ivDialogCloseButton"
  Then I wait for 2 seconds
  Then I press view with id "top_bar_btn"
  Then I wait for 2 seconds
  Then I press "Beginner"
  Then I wait up to 10 seconds to see "ACHIEVEMENTS"
  Then I wait for 2 seconds
  And take picture

Scenario: Signup switch directly to birthday wrapper
  Given I wait for "Login now!" to appear
  Then I wait for 3 seconds
  Then take picture
  Then I press view with id "email_btn"
  Then I wait for "Why do I need to provide my birthday?" to appear
  Then I wait for 2 seconds
  Then take picture
  Then I press view with id "choose_display_name_field"
  Then I wait
  Then I press view with id "choose_birthday_shadow_wrapper"
  Then I wait for 2 seconds
  Then take picture

Scenario: Signup successful
  Given I wait for "Login now!" to appear
  Then I wait for 3 seconds
  Then I press view with id "email_btn"
  Then I wait for "Why do I need to provide my birthday?" to appear
  Then I wait for 2 seconds
  Then I press view with id "choose_display_name_field"
  Then I enter text "android-acceptance-test" into field with id "choose_display_name_field"
  Then I wait
  Then I press view with id "choose_birthday_shadow_wrapper"
  Then I wait for 2 seconds
  Given I set the date to "03-02-1985" on DatePicker with index "1"
  Then I wait for 5 seconds
  Then I touch the "Set" or "Done" button
  Then I wait for 3 seconds
  Then I press view with id "next_btn"
  Then I enter a random email user "android-acceptance-test" into field with id "sign_up_email_input"
  Then I wait
  Then I enter text "qwerty123" into field with id "sign_up_password_input"
  Then I press view with id "next_btn"
  Then I wait up to 10 seconds to see "WELCOME"
  Then I wait for 3 seconds
  And take picture

New to Testmunk?
Testmunk automates mobile app testing on real devices. Create an account on testmunk.com, and let’s get started.

Testmunk automates mobile app testing

LEARN MORE

Read next


Create Your Testrun More Quickly

Posted by Michael Walsh on December 17th, 2014

Powerful logs and more android devices available

Posted by Martin Poschenrieder on November 4th, 2014