It’s a curious case, and a dubious honor to be sure, to be in Facebook’s position. To simultaneously have two of the worst rated applications, and two of the top ranked by number of downloads is a unique achievement. As of this writing, as reported by App Annie, Facebook is ranked at number 3, and it’s unbundled sister app, Messenger, sits at number 2. Facebook’s ratings in the review column for the U.S sits at 1.7 stars, the worst app by far listed in a top 30 ranked application. The second worst reviews average is Facebook’s Messenger app, with an average of 2.7. One thing is clear, for the past few months at least, Facebook for mobile has become the app everyone loves to hate, even while they can’t seem to avoid using it. For a company who takes 59% of its revenue from mobile advertising, their complacency with regards to this recurring review rating is surprising.
Facebook’s app rating (from App Annie)
This love-hate relationship is not entirely a recent development either. Four years ago, in Facebook Mobile’s toddler days, an expensive and very unpopular attempt to create the mobile platform as a HTML 5.0 web app resulted in a very public back-track and switch to iOS native development. For a long time, the ratings of Facebook’s iOS app improved, and until recently, held steady at roughly four stars. Throughout, Facebook’s social application download ranking has hovered in the top two slots, and has held the overall most downloaded ranking of between 3 and 5.
It is a known fact that if something is wrong, there are many more people willing to complain about it than there are who will offer kudos for a product that does what it should. A lack of complaints from a large portion of the user base is not a sign of prosperity when the only comments received are negative. The numbers coming out to complain are called the vocal minority for a reason. Out of the 100 most recent reviews, based solely on U.S. numbers, ninety were rated 2 or below. Only four provided a perfect 5. A consistent average rating of less than 2 stars in the U.S. for several releases running is a mark of serious problems, regardless of how many are still using the product. A consistent average rating of 2 or less in nearly every country should be terrifying, even to a company as large and relatively unchallenged as Facebook.
Facebook’s worldwide app ratings (from App Annie)
By the Numbers
Perhaps even more startling than the number of complaints being reported is the variety of issues being reported. By grabbing the hundred most recent negative US reviews for the time frame of January 1st to January 15th from App Annie, I was able to break down the numbers into some basic categories. Here are the top five complaints from users of the iOS Facebook application:
Note: In the above numbers, there is significant overlap, because several users cite more than one of the issues noted.
- App Crashes: For simpler categorization, this includes difficulties in loading the application, and app hangs. A whopping 52 out of the 100 sampled negative reviews related to the fundamental reliability of the app.
This complaint did not show up as often in the hundred sampled responses but is commonly enough seen to warrant mention. Four users are incensed that Facebook publishes updates every two weeks, but does not list the fixes supposedly made. Another few do not mention the update policy, but don’t feel that repeated updates are fixing any of the problems reported.
Where is the Disconnect?
By no means is it easy to ship an application that is used by billions of users, and the people I have known who work or have worked there are unquestionably brilliant. However, the numbers speak for themselves, and the approach used by Facebook is currently failing miserably. Why? It is widely known (and recently lamented) that Facebook has a quick two week release cycle for updates (as per their release announcement and schedule on the app store) and internally, by a couple reports, releases a sprint every tuesday (Note, these sources may be referring only to Facebook itself, not it’s mobile counterpart and also may be out of date.) It is also well known that the internal philosophy at Facebook has been to ‘move fast and break things’, as per the famous quote from their founder, Mark Zuckerberg. While this motto has changed, according to an announcement made at last year’s F8 conference to “Move fast with stable Infra”, the change has not resulted in a better app for iOS user.
As stated, Facebook is generally close-lipped about their practices, but have shared tidbits here and there at conferences and other venues. According to a presentation by Alan Cannistraro, Engineering Manager at Facebook, there are typically 4 Facebook applications running on every employees phone: A local build, a daily build, a release candidate and the App Store release. There is a QA period for each phase, but it is unclear exactly how long. What is known is that each developer is largely responsible for his or her own code, and there is no official QA department. Rather, the entire company is used as a test-base.
Quote from Alan Cannistraro – “As you may know, Facebook does not have big QA teams…we believe that developers are responsible for their own code, and they’re supposed to write the tests to do that.”
Regardless of whether there is a two week QA cushion that the public does not really see, this is still a lightning fast development cycle for an app the size of Facebook, with so many teams and moving parts. The net result is added pressure on the developer doing his own QA, as well as a relatively short window to fix bugs when found. This is further exacerbated by the fact that Facebook reportedly builds “all iterative development in their master code branch”. This means potentially hundreds of unconsidered use cases (Facebook does mitigate this by building on/off functionality into every component/feature they build) in every release.
To execute an extremely short sprint cycle on a large and complex application is heavily reliant on process and tools. Recent evidence suggests neither are proving adequate for Facebook.
As reported by Alan Cannistraro, some of the primary testing tools used by Facebook are:
- Snapshot unit tests, with kaleidoscope to verify pixel accurate views
- Testing by employee base -multiple builds on each device, different icons for each with build numbers burned in for easy reporting
- An Internal Settings development Dashboard allowing developers to:
- toggle features on and off
- change parameters
- trigger events
- Watchdog Timer – A “max-priority thread” pings the main thread every ‘X’ seconds. Any block of code that exceeds this ping interval is logged automatically. Similar to Apple’s Watchdog timer, but Facebook is able to set the interval to a higher degree of sensitivity(down to 1 second).
- Shake Report – developers or other employees are able to ‘shake’ their device to auto-send a bug report including annotated screenshot, logs for x seconds, network and crash logs.
Working with various fast growing companies, I have seen that developers and QA in such organizations also need the capability to have access to a large set of devices in different configurations that they can run large-scale testing automated on. They need to not only be able to quickly catch regressions, but also monitor and benchmark performance implications of their latest pushes. Using simulators for initial testing runs, as I believe Facebook is currently doing, does not catch enough issues.
The absence of a specific QA function may be hampering the company as well, because the average user (even a developer user) is not in charge of large scale regression testing, no matter the philosophy. He is typically testing to validate his work or that of his colleague. This leaves far too large a margin for error in each release, particularly given such a short build and QA cycle (such as it is).
From the reviews, a few possible problems can be surmised:
- Facebook may not have the tools and processes in place for such a short release cycle.
- A two-week release cycle appears (so far) too short to achieve a high quality for the number of features shipped for an app of Facebook’s size.
- Facebook is attempting to push too much into each release.
- Facebook is doing a horrendous job at communicating with it’s users.
- Possibly all the above.
Changing any one of these factors is very likely to improve Facebook’s ratings and their public image. But the last one is almost inexcusable for a company built upon communication. Users are flabbergasted at not being able to see the fixes being released, and therefore are all the angrier about the crashes and failures. I personally believe that updating users on exactly what is going into the builds being released may do more to mitigate the poor ratings than any other single factor. After all, it is far easier to hire a technical writer to provide succinct release notes and possibly some fixes to known issues than it is to hire more developers to fit more fixes into a sprint.
I have seen many apps and companies alike succeed with a short build cycle. There is no reason even an app as large as Facebook could not succeed with one as well, provided the right tests and processes, and clear communication about what can be seen in each release. I believe that ultimately people will get used to the fact that the messenger app is a separate app and poor reviews on this will become fewer. Ultimately, it makes sense for the user to give the best experience in an isolated app. However, even on the Messenger side of things, communication and proper testing is key. If you unbundle functionality from a larger app, you absolutely cannot afford to break it.
Messenger having problems too.
In the same way, it makes sense to limit what Facebook has to do, and has to do well. If this is the plan that Facebook is attempting to follow, the short release cycle makes sense. However, without a clear sense of direction, and clear communication of this plan to its users, negative reviews will likely continue.
At the end of the day, it is all about ensuring that the user not only continues to use the application, but has an amazing experience when he/she does. The one thing that Facebook can’t afford to “break” in their efforts to “move fast” is the trust of their customers. My motivation for writing this article was a statement made by a friend. “Wow…Facebook has 1.5 stars, makes me wonder if I should even download the app.” This led me to ask other questions, such as those I attempted to answer in this article. Other questions, such as “What impact are these negative reviews having on their mobile revenue stream, considering the expected growth in this market?”, can only be answered by time. Mobile advertising in April 2014 made up 59% of Facebook’s total ad revenue, up from 30% one year prior. This means that even if Facebook is only standing still in the mobile market as a result of these reviews, they are still losing ground.
|About the author:
Martin Poschenrieder has been working in mobile for most of the past decade. He started his career as an intern for one of the rare German handset manufacturers, years before Android and iPhone were launched. After being involved in several app project and realizing the pain of mobile app testing he started testmunk. The company provides automated app testing over the cloud. Follow Martin on twitter