One regularly asked question received from our customers and prospective customers is what tools we use for bug tracking, and what some of the best tools are. This is a difficult question to answer, as it can sometimes seem that there are almost as many solutions out there as there are developers to use them. Indeed, in asking about bug tracking tools, many of the solutions presented were not bug tracking tools per se, but tools that featured bug-tracking as part of their repertoire. Others tools mentioned were narrower in focus and specifically focused on app crashes. So one problem is that few developers have worked with enough of the various options to definitively state which is “best”. The other problem is in defining what a bug tracking solution is. If you use an excel spreadsheet, and name it Bug Tracking, then technically, isn’t that your solution? In addition, a given solution might only be “the best” for that particular person, or that particular organization, because it happens to coincide well with their organizational processes.
Spreadsheet Issue Tracking
With that said, tracking and reporting bugs is a very important part of the development cycle, and how it is handled can greatly affect the outcome of your release, if not the app itself. Because finding the right solution for you or your organization is so important, we decided to make the attempt to help in informing your decision, if not outright providing you an answer. In order to deal with this very subjective question, we asked co-workers, developer and QA friends, and random respondents on various chat and bulletin boards which products they had used, and which they preferred, and why.
In order to discuss “the best solution” we first have to define what a bug-tracking solution (sometimes referred to as an Issue Tracking System or ITS) needs to do. An ITS generally provides the user with a way to report an issue, track progression towards its resolution, and know who is responsible for resolving the issue. A solution for a small development team might be as simple as a spreadsheet with details on the bug, columns showing predetermined status/progress levels, and an assigned resource. This spreadsheet will quickly become unmanageable, however, if the project builds in complexity, or more team members join the fray. If managing multiple projects, or a complex program, then it is best to begin thinking about making an investment in an ITS (bug-tracking) solution.
One of the key factors to choosing a bug-tracking tool is how obtrusive it is. How much extra effort does it require to maintain, and does documentation of the issue take as much time as fixing it? A good bug-tracking system needs to fit seamlessly into your process, and ideally, allow you to track progress while working on it, rather than before or after.
Multiple Solutions For Smaller Teams
In some cases, a bug-tracking solution may not be a bug-tracking solution at all. Some teams, notably those who use lean development, employ automated testing as a form of bug-tracking. Lean development is not truly a methodology in and of itself, but a philosophy applied to methodologies (for example, Agile and Lean are not synonymous, but Agile has Lean elements) – it is focused on reducing waste or wasted effort in a given system. In this case, lean development would argue that by finding and fixing the bug, you are essentially tracking and managing its lifecycle organically. When a bug is found, automated tests are written to reproduce it, both fixing and documenting the bug. When the code and test both are checked in, the bug is documented, and alerts the team to the bug if it should occur again. In this manner, the team can “fix and forget” the bug, without unnecessary process steps. For those interested in experimenting with such a process, Testmunk might prove helpful. Not only does it allow you to document the tests, but also screenshots are automatically provided to allow your team to visually determine if the bug is fixed across all devices. Typically, however, this bare-boned process only works for small, very disciplined teams, and does not scale well with larger development projects. The more bugs found, and the more people involved, the more you will come to realize the need for a more traditional bug tracker, in order to prioritize.
Some of those we spoke to told us that they used Github’s native functionality in terms of bug-tracking software. Github’s integrated issue tracking is surprisingly robust, offering flexible filtering, keyboard shortcuts, security (limit access to teammates on private repositories) and accessibility for public repositories.
Github Issue Tracking
“Our team uses Github for SCM, pull requests/code reviews and bug tracking. Issues integrate well, is reasonably lightweight (you’re not spending more time adding meta-data than working!) but powerful enough to let us tag/categorize/filter issues.” -Anonymous QA Respondent
Because Github is one of the most commonly used code repositories, familiar to most developers, processes often revolve around the product. For this reason, integrated functions are often adopted more quickly than supposedly better third party add-ons and tools. In sticking to the familiar tool, they are in fact cutting potential wasted time from their development cycle, and not just by avoiding additional training requirements.
“We also tried a couple things that try to build on top of Github like blossom and Waffle. They weren’t *bad* but it did not seem like the integration was tight enough. You either had to get everyone to abandon Github’s issue tracker and work exclusively in one of these 3rd party tools or you wouldn’t have things tagged right and would lose the value of the tools. In the end using Github directly with less features was just a better solution.” -QA Manager
While familiarity and streamlined process can be key factors, even at the expense of useful features, Github was not the most common or popular choice presented. For smaller teams, however, simpler tools with less features was a common theme. One other popular choice was Crashlytics. When asked to expand upon his reasons for promoting Crashlytics over HockeyApp, a friendly Reddit respondent replied with the following in depth analysis:
“Free, and dead simple to use. You just click on button on the IDE plugin and you have the library installed and configured in your project. You can also upload apks directly from the plugin. You can include Twitter SDK integration.
It also provides a really helpful dashboard. It gives you detailed info about crashes, how many users are affected, version, devices. The crash information provided is much more detailed and better presented than hockey app, which has a rather ugly dashboard. It also provides a live chart showing the number of users currently using your app. To sum it all up, it is as the name states, crash reporting and analytics under one roof.
As far as problems, well it is a closed source product, and when I first tried it, had a tendency to increase build time a fair bit…at first went from 1 minute to 10 minutes. However, I think this is resolved now. Hockeyapp is open source, and also was pretty simple to use, and had a smaller footprint on the final apk. Overall though, prefer Crashlytics.” -Reddit Respondent
Another respondent, coming from a specifically iOS perspective, had this to say:
“I’m usually using Crashlytics. It’s part of Fabric now. It’s the easiest way since Apple acquired TestFlight to distribute beta builds, and it also gives you bug tracking, real time analytics etc. I’ve been using a couple of other solutions too, like InstaBug and such, but Crashlytics seem to me like the most polished and streamlined drop-in solution for it.” -Anonymous iOS developer
It should be noted that Crashlytics is not considered a true bug-tracking tool by some, being more focused on crash reporting, rather than different types of potential bugs. Still, it came up more than once as a bug-tracking tool, and one with positive reviews. It is obvious that defining what qualifies as a bug-tracking tool is nearly as subjective as determining the “best” solution.
Redmine was another tool recommended in a similar manner, “free and easy to use”. At Testmunk, a fair number of respondents expressed a similar preference for Trello, when it comes to smaller teams.
Trello as Bug Tracker
“I really like Trello to keep all our tasks and ideas. But I think it is more convenient for a small group of engineers.” -QA Engineer
“There is also Mantis. But, it’s a heavy tool. It really depends on the context / development process. I’d start with Trello or even things like Github Issues.” -QA Engineer
“The companies where I used to work tended to each have different tracking platforms…for example, Mailru Group used Trello as a bug tracking system for smaller projects like mobile apps and so on. At Moka5 Inc. we used Wiki Trac, and I can say that this was one of the most inconvenient platforms for BTS (Bug Tracking System) that I know of. I would suggest Trello as an optimal and not so expensive solution for this kind of thing in a small company.” -Automation engineer
As you can see, simpler and more basic solutions are recommended for smaller teams. Trello, Github, and Crashlytics received positive reviews from a few different sources, when it came to the small team dynamic.
Jira For Larger Teams
As stated, while popular, Github was not the most popular selection amongst those asked. That honor belonged to another program that was not strictly a bug tracking tool. Indeed, it is a tool more commonly associated with project management. It does, however, (like Github) include fairly robust bug-tracking components, as well as plug-ins. What is interesting about this solution is that it was recommended for some of the same reasons as the above mentioned tools – easy to use, inexpensive (but not free), good reporting, and not obtrusive to the project process. The product that came up more often than any other seemed to be Atlassian’s JIRA.
JIRA as Bug Tracker
One respondent listed the following advantages for JIRA:
- Tons of different plug-ins (Zephyr, Crucible, etc)
- Kanban/Agile Boards
- Basic (drop downs)/Advanced (Queries) searching
- Time tracking
- Fast setup
- Clean/Fast UI
- Lots of reporting tools
One key feature then, that tools designed for smaller teams might be missing then, is extensibility. Github is known to have several available add-ons as well, but perhaps not as well developed as Atlassian’s.
“Jira is simple to start with but comes with everything you might want to use later on.” -Android developer
“Sure, it’s quite clunky sometimes, but the customizability is second-to-none and the plug-in selection is excellent.” -Reddit Respondent
Another strong reason for Jira’s popularity seems to be an intuitive user interface and dashboard. Jira manages to inject itself into the same conversations as Trello, Github, Redmine and other simple and intuitive tools. Of the solutions usually billed as enterprise ready, JIRA seems to have the most positive input about it’s usability.
“JIRA is pretty good for bug tracking. I particularly enjoyed the customizable dashboard to easily view different types of bugs and progress.” -Android Developer
“If you want some sort of workflow (agile/scrum) work along with your bugtracker (or at best have in integrated), then you want more than just the Github bugtracker. We have worked with Polarion, Redmine, Mantis and Trac, as well as Jira. I too would recommend Jira.” –Slack Respondent
“For me, JIRA is a great bug tracking tool; I’ve used it in a small business (OnDemand version) as a task management tool, and in my current testing role in a web agency.” -Reddit Respondent
Users recommending or mentioning JIRA tend to agree that it is a fairly robust solution, designed with larger teams and a need for a set process in mind.
“In the past I have also used Jira. It’s pretty great if you’re forced to hand access to your tracker over to users who will not follow directions. You can force your issues to follow a specific path so nothing gets bypassed.” -Reddit Respondent
Because JIRA is a larger tool, with a heavier footprint, it is not universally loved.
“JIRA is OK, but in the wrong hands can turn into a mess due to being so configurable. We have some workflows set up that looks like the London Underground map.” -QA Manager
One user, who expressed that a bug reporting system should maybe be considered on two fronts, bug reporting vs bug prioritization, stated that:
“JIRA often crops up as one “does-both-option” but I think it’s too heavily tuned towards classic bug reporting, and the UI is horrible for team prioritization or usability.” -QA Tester
In addition, it is sometimes not considered as a bug-tracking tool, because it has several other functions. In a way, this additional functionality can work against a product. When mentioned as a bug-tracking tool, some argue that it doesn’t qualify. However, as mentioned above, a solution should not be discounted due to a narrow definition.
“Perhaps due to our workflow, I view Jira as primarily a project management tool that also can be used to track tasks and bugs. When we first adopted it we were using Teamtrack as our change management /defect tracking software, which was integrated into Perforce and JIRA to allow for project management, bug tracking, and change management. We’ve since migrated to primarily using JIRA.” -Slack Respondent
Of interest here is that the team in question experimented with a multi-solution system, but soon dropped several of them in order to simplify. However, while the respondent did tell us that JIRA became the primary tool, he was not convinced it was the ideal solution.
“Jira is fine, and the zephyr plugin allows for an acceptable test planning and execution solution, but it does seem less than ideal to me…there are other solutions to explore, depending on the size, needs, and budget of the company.” –same Slack Respondent
Again, the right solution for any organization is entirely subjective, and depends on a multitude of factors, including those mentioned above. Sometimes, the right solution will not immediately be loved, either. Some of the solutions named earlier with positive reviews, such as JIRA and Crashlytics, were not immediately loved, may not have fit some users definition of a bug-tracking tool, nor did they get universally positive reviews.
Pivotal Issue Tracking
Other tools mentioned for larger organizations included Pivotal, which received mixed reviews. One respondent preferred it to several others, including JIRA, but admitted it had not initially impressed him.
“I’ve used Redmine, Jira, and currently pivotal tracker. (I) can’t say that I love any of them, but I’d pick Pivotal if I had to choose because you always have the full view of your stories/bugs/tasks. It’s very different from Jira and Redmine, took me a while to get used to it.”-QA Manager
Another respondent had this to say in response to the above Pivotal recommendation.
“I have my own personal issues with pivotal tracker but that’s mostly because I’m a grumpy agilist and don’t agree with how the pivots view teams.”-Anonymous Developer
In other words, Pivotal behaves differently than JIRA, Redmine and some other solutions, and is found initially off-putting because of it. If one reads between the lines a little, a familiar user experience seems to be a fairly strong reason to recommend a solution. This comes back to the obtrusiveness of the system. Users do not truly want the most innovative system in bug-tracking, if it stands apart from the development process. If it requires a large learning curve to implement right, or an existing process has to change drastically for it to work, it is difficult to get buy-in. Sometimes, especially when you are starting out and just searching for your first bug-tracking system, familiarity might be the biggest factor in your decision. If six of your ten developers have used JIRA, and are familiar enough with it to implement it, and work it into company process, that might be the best solution. If it is not the chosen, or “best” solution, chances are that the solution chosen will be “like JIRA” in some significant way.
If your team is small, or projects are relatively small and simple, simpler solutions like Trello, Crashlytics, and Github’s native features seem to do the job fairly well. Of these solutions, Crashlytics and Trello seemed to be the most popular. If looking for a solution that is extensible, and can be customized to grow with your organization, the top choice seems to be JIRA. Jira is also about project management, and can be used to push developers towards a unified process flow, in which bug-tracking and resolution is an integral part. Even if it did not get a glowing review, most of the developers spoken to had used it, and knew how it worked. This familiarity extended not only to how the solution is used, but to the tools and add-ons, such as Zephyr. This familiarity in in itself a valuable commodity. However, other solutions (such as Pivotal) have made some headway, may prove a fit for your organization as well, provided users are willing to adapt process.
Sometimes, the right fit isn’t out there, or the right fit necessitates more than one product. Several respondents mentioned that they had used several tools, but had ultimately decided as an organization that a custom tool felt better to them. Others used plug-ins to extend the capability of a mainline product. JIRA users often mentioned Zephyr as a helpful plugin. Github users touted that the API was easy to work with. The right solution for your team may not be any of the above solutions on their own, but rather a combination of solutions. The right solution might be one of the above-mentioned solutions, but implemented in an entirely new way. It is about finding the right fit for your team, and for your organization.
Tell us about your bug-tracking solution, and what you like or don’t like about it? Have you implemented a basic system in a new and exciting way? Share with us!
|About the author:
Martin Poschenrieder has been working in the mobile industry for most of the past decade. He began his career as an intern for one of the few German handset manufacturers, years before Android and iPhone were launched. After involvement with several app projects, he soon realized that one of the biggest pain-points in development was mobile app testing. In order to ease this pain, he started Testmunk. Testmunk is based in Silicon Valley, and provides automated app testing over the cloud.
Follow Martin on twitter