Hybrid vs. Native Apps
Many of our clients take a hybrid approach to app development. By hybrid we mean using technologies such as Phonegap, Cordova, Appcelerator Titanium, and Sencha as part of their technology stack. Hybrid apps can benefit from cross platform mobile testing.
Companies choose the hybrid approach for several reasons. A major factor is the ability to release on multiple platforms at the same time. The decision of whether or not to make a hybrid app comes down to the skill sets of the various team members developing the app. There are also some downsides to going non-native, such as the possibility of limited features, rendering and responsiveness issues, or crashes on certain devices.
Our customers have asked us about the best testing practices for identifying and addressing these challenges. Our response is cross platform mobile testing. Since hybrid apps are developed from a common code base, it makes perfect sense to have functional UI test cases that can be executed on multiple platforms.
What are cross platform tests?
“Cross platform” refers to the idea that the same tests (scenarios) can be executed on multiple platforms and/or device types. Imagine your test framework being able to test your hybrid app on both Android and iOS while remaining robust, lean, and easy to maintain. Below we will discuss cross platform mobile testing for not only Android, iOS, and Windows Mobile, but also different devices, like tablets and phones.
In this article we will show you the benefits of cross platform tests, when you should consider them, and what the best practices are regarding them. We will use the Calabash framework as an example, which in our experience is very good when it comes to mobile test automation.
Benefits of cross platform app testing
The same scenarios can be executed on different platforms
Let’s stick to the Android and iOS example here. If you find that the following scenario describes the same functionality on both versions of your app, why not have it written only once and executed on both apps?
Scenario: As an existing user I should be able to login and see the Home screen Given I am on Welcome screen When I login Then I should see Home screen
Test logic is uniform and can be reused
Let’s break down the When I login step from the above scenario:
When(/^I login step$/) do @app.welcome_screen.email.insert_text(‘firstname.lastname@example.org’) @app.welcome_screen.password.insert_text(‘test’) @app.welcome_screen.login.touch end
In all likelihood, the same test logic will apply for both your Android and iOS apps. Consider how much time we save by writing the scenario and step definitions once, instead of once for each OS.
In case a step differs between devices or platforms, you can always keep them separate. For example, the above scenario may apply for phones, but you may have a different UI on the tablet version, such as users needing to check the terms of service before pressing the login button:
When(/^I login step$/) do @app.welcome_screen.email.insert_text(‘email@example.com’) @app.welcome_screen.password.insert_text(‘test’) @app.welcome_screen.terms_of_service.check @app.welcome_screen.login.touch end
Less code to maintain
Less code not only means that you can script your tests faster, but also that you will save time on maintaining and updating them.
We can break up acceptance tests into 3 layers:
- Business language scenarios written in a declarative way (describing what you want)
- QA step definitions written in an imperative way (details how to do it and what to assert for)
- Screen objects, the interfaces used to interact with an app
Previously you may have had to write 1 or 2 tests individually, for every platform. Now, if your scenarios are the same for different platforms, you can reuse them. The only layer that will differ is #3, your screen objects. We will explain this in greater detail in a minute.
What apps can be automated this way?
If your app looks and behaves similarly on iOS and Android, or you don’t have different user experiences between tablets and phones, there is a great chance you can benefit from cross platform mobile testing.
It’s also great for hybrid apps, which are getting more and more popular, since these apps have a consistent look across different platforms.
On the other hand, if your apps are significantly different, it’s better to have dedicated tests covering their unique functionalities. In such situations you should consider having only parts of your app tested the cross platform way. The same applies when the apps have different release cycles; it may be the case that some features are introduced on only one platform at first.
Keep in mind the 3 layers we mentioned. It’s not always a good idea to make as many of them as uniform as possible. Sometimes it’s better to have different step definitions for the same scenarios across platforms.
Consider the WordPress apps. Despite them both being native apps, they have very many similar features, and most of these features could be defined by the same scenarios.
The scenarios below can be used as an example of a cross platform test. This test applies for both the Android and iOS versions of the app.
WordPress for iOS (left) and WordPress for Android (right)
Scenario: “Freshly Pressed” should contain an article When I am on the Freshly Pressed screen Then I should see an article
WordPress for iOS (left) and WordPress for Android (right)
A more imperative scenario about login in feature:
Scenario: As an existing user I can log in Given I am on the Login screen When I enter “firstname.lastname@example.org” into “Username or email” And I enter “test1234” into “Password” And I press “Sign in” Then I should be logged in
How do we do it at testmunk?
At Testmunk we use the Calabash framework, together with the screen object approach, to deliver robust tests for our client’s apps.
Screen Objects serve as the interface to an app. Have a look at the following test step, which enters “email@example.com” into the email field on the screen:
Is the app being tested.
Here we refer to the first screen of the app, which contains the login form; we called it “Welcome screen”.
Email is an input field in the Welcome screen.
insert_text is a method used to fill in a field; n this case it fills the email field.
Imagine every screen of your app described this way. This is called the Screen Object or Page Object approach. Your test code is easy to use, organized, maintainable, and hidden from the business logic layer.
Ok, back to cross platform mobile testing. Having screen objects be cross platform is easy. All you really need to do is implement custom screen objects depending on the platforms you need to support, e.g. Android phone, Android tablet, iPhone, iPad, etc.
The details of how to touch a button or swipe a screen on Android and iOS are hidden behind the screen objects.
By having the same interface (method names) like “insert_text” for the input field (@app.welcome_screen.email.inser_text), the step definitions can look exactly the same:
When(/^I login step$/) do @app.welcome_screen.email.insert_text(‘firstname.lastname@example.org’) @app.welcome_screen.password.insert_text(‘test’) @app.welcome_screen.login.touch end
Cucumber (Calabash) provides a parameter called “profile” (-p) that can be used to load different page objects depending on the platform / device type you test on.
calabash-android -p android-phone cucumber -p iPad
That parameter will search for the corresponding configuration in the config/cucumber.yml file in the root of your features/ folder. Here is how that file could look:
android-phone: -r features/support -r features/step_definitions -r features/android/phone iPad: -r features/support -r features/step_definitions -r features/ios/ipad
android/phone/ and ios/ipad/ contain the screen objects for each platform.
When executing on our cloud of devices (encompassing multiple platforms and devices), you don’t have to worry about your config file. We will make sure that everything works smoothly and your app is ready for all markets.
Cross platform mobile testing is not for everyone or every app. However, cross platform mobile testing can be a powerful tool that saves time and effort, especially as hybrid apps become more prevalent. Many apps have similar user interfaces across operating systems, and these similarities can be harnessed to create tests which work on both Android and iOS (and even Windows Mobile). Cross platform mobile testing is something we embrace and which is becoming ever more important in modern testing. As devices and operating systems continue to fragment, it is imperative to have test scripts which are lean and yet strong, and embracing cross platform mobile testing is one way to help ensure tests remain easy to maintain and update.
What are your thoughts on cross platform mobile testing? Let us know on Twitter or leave a comment below.
|About the author:
Lukas is a QA Engineer at Testmunk helping some of the biggest iOS and Android apps to establish large scale automated test suites. Before discovering his passion for automation, he worked several years as a mobile app developer. He is passionate about agile and lean methodologies and in his free time he loves to ride his motorbike.