This is a discussion topic started by senior test engineer Chakradhara Pernumarthi on the Software Testing & Quality Assurance group in Linkedin; one of the most popular business-oriented social networking websites.
This discussion is one of the most popular currently on the group discussion board, with 120 comments from group members, which primarily consisted of software testing and quality assurance professionals.
The discussion was started in April 2010, probably inspired by another popular discussion in the same group ‘Bug Tracking Software – What‘s your tool of choice?’ started by QA analyst Domonic Prince. Domonic had started his discussion in October 2008, and had generated about 260 comments, before mellowing out in March 2010.
Both these discussions have given a unique opportunity to gauge the popularity of tools, both open source and commercial ones, among the testing community.
They can also be considered as a great resource for valuable feedback on what the specialists in the feld look for in such tools when trying to choose one.
This article provides a roundup of the most popular tools that anyone evaluating bug tracking and test management software should consider with insight and tips from the experiences of testing professionals.
Introduction
Although open source software tools will be the primary topic of discussion, commercial tools will also be discussed in this article.
First, the list of open source software tools that were recommended by the members of the group in Chakradhara‘s discussion on open source software will be discussed.
Then we would like to look at the software recommendations from Domonic‘s discussion, which included non-open source as well.
The reason for this is that most people evaluating bug tracking and test management software are going to be looking for feature sets and functionality that are available in commercial tools.
Hence we will also discuss what the most popular commercial software is, what they ofer, and why they are considered for test management and bug tracking.
If we can consider the discussions as a survey, figure-1 shows the popularity of the diferent commercial tools available, including commercial and open source software.
The percentages inferred from the discussions seem to match the results published from survey Open Source Developer Report 2010 conducted by the Eclipse community.
Atlassian‘s JIRA software was popular among 16% of both the Eclipse community report and linkedin group members.
TRAC was popular among 10% in the Eclipse community report while only 2% of the members preferred it in linkedin group discussions. In the discussion, Mantis was popular among 11%, which was quite close to the 9% observed in the Eclipse community report.
The most popular bug-tracking tool that members of the group preferred was Bugzilla, attracting 22% of them.
15% of the members preferred HP’s Quality Center (Formerly Mercury Test Director), while 6 % liked Rational ClearQuest. Microsoft Team Foundation (TFS/VSTS) and FogBugz were each popular among 2% of the linked in members. 3% liked Seapine TestTrack Pro.
We shall now see what the members liked and disliked about the various products, which we shall broadly classify under commercial software and open source software.
Open Source Bug Tracking Software
The figure-2 shows the popularity of the diferent open source bug tracking software tools discussed in the linkedin group discussion by the members.
Bugzilla
Bugzilla was the most preferred bug tracking software among the members of the discussion group convincing 60% that it is the most ideal open source software.
Jack Mangan says that I know Bugzilla isn‘t perfect, but it does the trick. There are a lot of customizable reporting features and its UI isn‘t too difcult for the non-techs to navigate. Nice price too.
Amjad Kayani however thinks that although it falls short in terms of reporting capabilities, it can work fne for smaller projects.
Daniel Emerald has a great tip that there is a Windows-like overlay called Deskzilla that they are starting to use for those that want that form of UI.
Although it is the most popular tool, there were some members like David Baer who found Bugzilla is by far the worst. He thinks that setting up bug fow rules and management of individual bugs is very cumbersome. He also found searching for bugs was unnecessarily complicated an inefective.
Akashdeep Awasthi summed up the features generally considered by most members when they want a bug tracking tool. He says; if you really want to go open source for your bug tracking and test mgmt, then no doubt „BugZilla“ is the best one to go, due to following reasons:
- Easy to setup and easy to understand.
- Easily integrates with TestLink which is a test management tool.
- Very test friendly.
- Contains almost all the features found in commercial products.
Rupali More suggested that Bugzilla and Test Runner or Bugzilla and Testopia can be used for Bug tracking and test management. Integration of Bugzilla and Testopia works really well. Bugs can be mapped to the test cases directly and it becomes easy to track your work.
Mantis
Mantis was the next popular open source bug tracking software, preferred by almost 30% of the members of the discussion group.
Shripal Shah says that Mantis is and open source tool with enough user-friendly functions having capability to engage any time of project size and bug count.
Apart from other tools available online, Mantis can be hosted on your own server with very little know-how of PHP and MySql. He has found that Mantis can be made operational within no time and time to learn is very low.
With custom feld creation facility, he has also been able to use the tool as a project management tool for some of the midsized projects. He sees no reason in spending a lot of money if you get everything to track and monitor bugs using open-source tools like Mantis.
Todd Lawall considers Mantis’ clean web based interface, ease of setup, minimum resource requirement (could run on an obsolete desktop if needed) as a big plus. He also points out that it doesn‘t require any particular platform for the client, only a browser, and no fancy scripting, so even smartpho-nes can handle it.
Rajat Prakash thinks that Mantis is good for the testers as it provides a simple enough interface for defect entry. But he feels that from a manager’s perspective JIRA is much better as it provides the summaries, trends and reports that are needed to manage a project efectively.
Francis Law summed up the Bugzilla vs Mantis debate thus, Both Bugzilla and Mantis are very popular, but none of them has a clear win over the other. Some people comment Bugzilla is very powerful but difcult to setup and customize. Some people comment Mantis is not robust enough.
I use Mantis and like its ease of setup/use and ease to customize. If you are a single person team, you can even use the instantMantis that can be setup and ready to use within 15 minutes.
TRAC
Trac was a popular choice of open source bug tracking tool among 5.5% of the members.
Jason Galyon comments that if nothing else he enjoys its great source viewing (changesets, difs, comparisons between arbitrary changesets, etc.).
He thinks that Trac is extremely confgurable and extensible and the source control integration is fantastic. He uses it as a primary wiki and exports to Excel, Word, and Open Ofce as needed.
Hugues Bonin has used DDTS, Rational ClearQuest and Trac. His favorite of all is Trac: great for small projects, allow tasks tracking as well and it‘s FREE (open source). The community helps improving it with plug-ins.
Sumit Madan thinks that Trac helps to get the developers involved. It integrates easily with subversion and the IDE of the developer. This way the developer can have a Trac plug-in while checking in fxes and can add their comments.
This helps the QA team link directly to the code changed in the bug comments to see how invasive the change (and hence how extensive the testing) is. It is all coded up easily in Python and super confgurable – also not as heavy as Bugzilla or cumbersome to confgure as Mantis.
Enrico Oliva asks why not use Test Manager plug-in for Trac to integrate test management. Robert Evans explains that they use TestLink + Trac. Integrating TestLink as a test management tool was a good choice for their needs.
BugTracker
The open source bug-tracking tool BugTracker was recommended by about 3% of the members. Pankaj Goenka recommends the web-based bug tracker as it provides features like high confgurability, email integration, search, screen-capture utility and ability to add custom felds.
Sreenadh OG further explains that after a review of numerous available solutions he preferred BugTracker. According to him it is the only BugTracking tool written by Tester from Tester‘s (Bug Tracking tool user‘s) perspective I feel.
He says, Use it, review it and arrive at your own conclusion and please don‘t get mislead by the not-so good looks of the poor demo hosted in their site (this bug tracker is highly customizable and it is very easy to change its looks and feel by applying a simple custom CSS).
The awesome fact is that the whole product is developed by a single individual, and his future target list and future plans are also simply unique and herculean!
Open Source Test Management and Complete Solution Software
Figure-3 shows the popularity of the diferent open source test management software tools and open source tools that provide complete solutions containing bug tracking and test management, that have been recommended in the linkedin group discussion by the members.
TestLink
As an open source test management tool, TestLink had the maximum recommendation at 53%.
Like Srikanth Reddy Komandla , most found it easy to create and manage Test cases as well as organize them into Test plans using TestLink.
These Test plans allow team members to execute Test cases and track test results dynamically, generate reports, trace software requirements, prioritize and assign tasks.
The general consensus was, as Renuka Arunkumar points out that it possesses almost all qualities of HP Quality Center. It needs to be combined with a defect-tracking tool, like Mantis, Bugzil-la.
Renuka uses Redmine. Francis Law had a good tip that it even includes a very simple requirement management module, thus bringing it closer to QC standards.
Nitesh Agrawal, Rewati Potdar, and many others suggest that TestLink and Mantis are a good combination.
Testopia
At 17% Testopia is the next most popular open source test management software.
Oana Casapu recommends Bugzilla with Testopia plug-in/extension (for test management). Together they look like HP‘s Quality Center, although the reporting part doesn‘t look that nice as in QC, but the most important features are present.
One disadvantage that she sees with Bugzilla+Testopia is that it‘s not that intuitive for a person that is not involved in software development activity. Business people or clients that just want to see the testing progress may fnd it a little bit complicated to navigate. But still, she feels, this combination is the best open source test management and defect tracking tool.
Ameer Khoja agrees with Oana, but he feels that installation can be a little tricky. There is a reporting plug-in, which you can install on top of it.
Sarath Chandrasekhar thinks that Testopia, which is basically TestRunner integrated with Bugzilla, is the platform that comes very close to HP Quality Center in terms of documenting, organising, assigning and executing Test cases. It includes a Test Cycle History and a loosely coupled integration between the Defect number and Test case.
Redmine
Redmine is the next popular open-source tool at 12%.
Kiran Deram explains that it was developed on ruby on rails technology. He feels that it is a wonderful tool for Test Management and bug tracking as it can be customized. Valmik Darade thinks that it’s just like JIRA.
RTH
At 7.5% popularity, RTH is the next recommended open source complete solution software.
As Rizwan Jafri puts it, RTH-Turbo is a simple, open source, light weight, easy to use and learn Test Case management tool that supports Test Case to Requirements, Requirement to Release, Test Case to bug associations features.
Xqual XStudio
Xqual’s Xstudio is one of the newer tools that have gained the interest of about 6% of the members.
Eric Gavaldo gives a detailed analysis of its features. He comments that XStudio handles the complete life cycle of the projects: users, requirements, specifcations, projects (dev, testing, etc.), tests, test plans, test reports, campaigns defects.
It allows scheduling or running fully automated or manual test campaigns. It also provides some tools to measure the coverage of testing (i.e. traceability matrix); all this via a local installation or through the browser.
It can run manual tests but also automated ones (through the provided open-source launchers): QTP, AutoIt, Selenium, Ranorex, Sahi, TestComplete, TestPartner, Visual Studio, Beanshell, Squish, NUnit, PyUnit, JUnit, TestNG, Marathon, Perl, Tcl, simple executa-bles, bat fles, etc.
In addition, the SDK allows one to develop specifc launchers to interface with proprietary tests. It includes a very good and appreciated bug-tracking database but can also integrate with: Mantis, Bugzilla, Trac, Jira.
Commercial Software for Bug Tracking and Test Management
The figure-4 below shows the popularity of the diferent commercial software tools discussed in the group discussion by the members of the business-oriented social network LinkedIn.
Atlassian’s JIRA
Among the diferent software tools discussed, Atlassian’s JIRA was the most popular commercial software for Bug Tracking and Test Management. Amidst the commercial tools, 33% of the members preferred JIRA.
Most of the members echoed Ofer Prat’s comment that he enjoyed JIRA a lot because it is extremely usable, fexible productive tool, easy to maintain and customize, and least expensive. Generally, the consensus was that Atlassian is developing quite an impressive portfolio.
Many of the members, like Jim Weeks, highly recommended JIRA, since their view on bug tracking systems was two-fold. Firstly, it gives them the information they want to see and secondly it is easy and intuitive for QA engineers and development engineers to work with. He says, I‘ve yet to have a user complain that JIRA gets in their way.
Some members like Mike Bresnahan, mention that JIRA works well when you have many needs. He says that his company uses it for facilities, tech support, IT helpdesk, hardware engineering, software engineering, and RMA reporting and metrics. The work-fows are user customizable and performance is pretty good. The only negative is that the search facility is not very good.
Some of them are so thrilled with JIRA, like Nitesh Agrawal, that they fnd the best tool ever for Bug tracking is JIRA. He shares his experience thus; I would say, one should try this tool at least once; it’s freely available demo version. It is an excellent and very easy to customize.
In the mobile domain, more bug felds are required to defne a bug properly. We have customized it successfully and we are executing release management and defect prevention nicely.
HP Quality Center
HP’s Quality Center was recommended by 31% of the linkedin group members for commercial tools. Most of these members were of the same opinion as Kiril Strax, who thinks it is difcult to fnd another product that would match all the possibilities of QC (e.g. installation/confguration options, very versatile and easy customization, report/export system, etc.).
It also scales very well, so overall it‘s a better match when fexibility is a must, when you want a test/defect integrated solution, or large number of end users must be supported. But it has two drawbacks: the price, and the fact that the client side must reside on Windows (even worse: you are mostly limited to IE).
Members like Galith Nadbornik (Chriqui) enjoy the Quality Center features. Starting from the requirement defnition and the linking possibility between the requirements and raised bugs and created Test cases, it simplifes the follow up for testers and provides extensive reporting tools for management. Easy to use, very fexible and the Admin can create as many modules and classif-cation are other strengths.
Another member, Karen Keirstead adds that you can tie everything together from test conditions through defect resolution with all the notes and the dates – time, and person entering the information. etc. It makes the need for traceability matrix documents obsolete. Jonas Stenman echoes this by pointing out that the possibility of tracing between test cases, defect and requirements are a great beneft. Also the metrics part is very helpful.
Danny Hemminga summarizes the key features. It gives you grip and control on the whole IT and application lifecycle. From setting up your requirements, set up your test cases, test execution, adding defects and link them all at the same time.
For larger projects it gives you more than defect based testing, but gives you also the workfow in which every employee has to work. Results, history, root cause, test coverage, defects linked to test sets and requirements.
Some members like John Hantak did have some issues with Quality Center. He shares his experience thus; Approximately two months ago, my team started piloting HP Quality Center. So far, I‘ve been impressed. The performance is more than acceptable and reporting mechanism covers most of my needs.
So far, the only areas that my team is struggling with are:
- There is no interested party feld in defects created in Quality Center, so you have to manually enter in the names of team members that you want to notify regarding a defect that isn‘t assigned or didn‘t create the defect.
- User A can overwrite user B‘s comments.
- The reporting mechanism can only report on two dimensions (i.e. Functional Area versus Priority or Functional Area versus Status). The tool that my team used prior allowed you to report on more than three dimensions (i.e. the Y axis has the Functional Area while the X axis has the diferent Status/ Priority combinations).
Rational ClearQuest
About 12% of the members recommended Rational’s ClearQuest as the tool of choice among the commercial ones.
The tool was recommended by members like Sripad Raj because it is famous for user friendly UI (user interface) and multiple features which makes it most suitable for all types of projects.
Seapine TestTrack Pro
Amidst the commercial software, Seapine’s TestTrack Pro was preferred by 5% of the members.
Group member Stacy Lukaskawcez shared her experience; Test-Track links to Seapine‘s source control (SurroundSCM) and several other third party source control tools but with SurroundSCM you get a bidirectional link that allow defects to link to source code changes for complete traceability.
Another group member Susan Sutherland Pina, liked Seapine‘s test track – since it was easy to modify workfow, status, and various other felds to ft your processes and lingo. Reports were decent and they never had an issue with response time.
FogBugz
Among the commercial software, FogBugz brought 4.5% of the member recommendation.
Sharon Matthews shared her experience as; I‘ve used FogBugz for projects both small (150 bugs or less) and large 20,000+) and highly recommend it.
Easy to use, you can get email updates as changes are made to each case, great customization abilities, flters, wiki‘s, reports, project planning, has great screen capture tools (screenshot tool for testers, SnapABug for customers, TechSmith SnagIt for advance screen capture, FreshLog for agile project collaboration, and Bug Shooting for email or Skype), has a host of plug-ins, time management and tracking tools, and then a great interface for entering and tracking bugs.
It isn‘t too expensive, depending on whether you host it on your server or FogBugz server and how many users you need it for. Take a look at their website or ask me further–I‘ve used it a lot on a wide variety of projects.
Kirsten Anderson recommends that you should go for Fogbugz primarily if you are working in an agile environment. One of the advantages is that it is highly confgurable.
Microsoft TFS/VSTS
Microsoft could only attract about 4% of the group member’s preference, among the commercial tools.
David Silvan was one of the strongest supporters of Microsoft’s tool. He had very detailed comments on the advantages and disadvantages of using this tool as follows:
After using a variety of diferent bug tracking systems, my personal favorite is Team System (TFS/VSTS). As a stand-alone bug tracking system it isn‘t anything spectacular, but the real exciting part is how well it integrates with all the other parts of the software development process.
Its tight integration with programming, builds, and testing, make it easy to make its use an intuitive part of any software process. For example, a build automatically triggered as part of continuous integration can automatically generate a bug based on failed automatic regression tests.
This kind of integration saves a whole lot of work and involves developers much more in the test processes. Another example of tight integration is that bugs can be linked directly to specifc lines of code changes, since the code‘s version tracking is kept in the same system. This is very handy for developers.
While systems like JIRA are also great for including bug tracking as part of an overall process control system, he hasn’t seen them integrate as completely as Team System does.
As someone who had designed processes, this made a big impression on our choice when evaluating a dozen diferent systems. He found the system fexible and customizable enough to be able to be intuitively included in teams using waterfall, agile, SCRUM, and a combination of the above.
The one area they found lacking was an intuitive interface for customer support. But Microsoft came out with a scaled down web portal that worked really well, since they could limit them to not having all the other distracting features and keep them on just bug reports and individual requests.
Team Foundation Server 2010 (TFS/VSTS) does integrate with the full software development lifecycle, including project requirements (i.e. user stories), test plans, test cases, work items, source control, bug tracking, reports, continuous integration builds, workfow management, etc. all integrated in a way that is incredibly powerful.
In other words, when testers investigate a bug, they can drill down to not only exact lines of code that have been changed in code churn, but also see the test cases and requirements that drove the change.
Developers, when investigating the bug can see play-backs of exactly what the tester did that brought up the bug, plus captures of any exceptions and errors. Project managers can get detailed reports of things like bug churn, burn down charts, etc.
TechExcel DevTrack
Unlike Microsoft, relatively unknown tool DevTrack was able to attract about 2% of the group member’s preference, among the commercial tools.
Lou Pedron recommended it as a solid product with many features including both a Windows and Web-based GUI.
According to him, projects are easily customizable for any application, in minutes. It also integrates with version control systems as well as DevTest for test case tracking. Service & Support are apparently excellent.
David Baer cautioned that DevTrack is a great application but doesn‘t provide a lot of value over other products if you don‘t combine it with other tools from TechExcel like DevTest.
If you have a need to track test cases/suites and would like to tie them in with bug tracking software (and have the budget for it) then DevTrack with DevTest is a good solution.
Comparison of diferent software solutions
Jose Endara observes that US Government projects for some reason use (in general) Rational ClearQuest, and Commercial projects HP-Mercury Quality Center.
Mary LeMieux-Ruibal comments; I‘m another one of the JIRA votes. It was a tough conversion since I had been a Mercury automated tool specialist for many years and heavily utilized Quality Center.
When completing a return-on-investment analysis on issue tracking software for my current client site, JIRA surprisingly came out leagues ahead of the competition.
The benefts:
- Customizable workfows, issue types, screen schemes and mandatory felds
- Open source
- Easy to use / low training overhead
- Cost for an unlimited license is a fraction of the cost of other issue tracking tools charge
- The plug-ins for ‚fsheye‘ for code traceability and the ability to link into Confuence have eliminated the need to keep a separate requirements trace matrix
My current client site has 875 separate projects running on a single instance of JIRA. We have over 9,000 concurrent users, 300,000 logged issues and the maintenance of JIRA is divided between a team of four persons.
I admit we have faced a lot of issues due to the sheer size of our implementation but I cannot imagine how Quality Center or ClearQuest would hold up to the abuse we‘ve put JIRA through. (Not to mention how much 9,000 concurrent users in Quality Center or ClearQuest would cost!)
Carolyn Ford opines that you need a tool that can incorporate the following: Requirements, Project Planning, Defect tracking, Test Planning, Test execution, and metrics to report on those areas in detail. You also need a tool that as „Couples“ these areas together, like for Requirements Coverage, Test cases associated with defects, and test coverage.
From a cheapware (Not freeware) option, JIRA ofers both defect tracking, project management, issue management all tied into one tool so you can have requirements traceability.
HP QC is great if business analysts use the requirements portion and testers use the defect portion, but it does not have a project-planning portion. Defect tracking is good in Bugzilla, but test planning is a miss there. Rational‘s suite of goodies, great if the organization has the bucks, same with HP. TestTrack pro also misses in the Test planning department.
Vlad Olteanu comments that the choice is clearly Bugzilla if you are aiming for a freeware solution. The table reporting system is good, and it‘s easy to read and administer. Add Testopia for project management capabilities.
If you are willing to pay and you need to manage/overview a lot of projects/project sections, go for J IRA. A correctly installed and confgured J IRA instance will allow you to see everything you need in just a simple look at your dashboard. He has also used Mantis and TRAC, but preferred Bugzilla over both in terms of functionality, referring to how easy it is to overview the entire project, see how debugging and bug reporting go along.
Marjie Carmen would frst recommend that you understand the needs of the system, like who needs the information and in what format. What types of reports will be useful for each of the stakeholders internally and if you have to provide reports to people outside of the company i.e. auditors, clients.
He chose Bugzilla when creating a new QA department, as it was open source and scaled well. He has used Rational, HP and Com-puware solutions, but feels they were rather cumbersome and expensive for small – mid-size frms.
Ideal only if you‘re going to implement other tools they have utilizing the shared business knowledge across the tools. He feels that J IRA has amazing confguration and reporting functions but still, as with any system, you have to be committed to ftting it into your culture.
They needed even a full time JIRA administrator as they continued to grow the tool for uses outside of defect/change management.
Conclusion
William Shafer has summarized the list of bug tracking tools discussed in the group. The list is at http://waysysweb.com/qa/ bugtracking.html
Caroline Ford recommended the link http://en.wikipedia.org/ wiki/Comparison_of_issue-tracking_systems for comparison of various software tools available for bug tracking.
Considering all the diferent tools and features, selecting the best bug-tracking tool is an interesting process. One way to go about choosing bug-tracking software is to frst list the features you need and prioritize them. If you already have an experienced software engineering team, borrow from their experience on listing the tools and/or features they would like to have in a bug-tracking tool.
Consult with other stake holders if you need an integrated solution for requirement gathering, project management, and so on. Discuss with management what type of reports and metrics are essential.
Consider the tools that meet the requirements, consider whether the tool can be customized to suit your needs and then match them with the budget that you have. Explore the possibility of trying out the tools you have shortlisted to evaluate the weaknesses, and revisit your requirements and shortlisted tools.
HP Quality Center is one of the most revered tools that most software testing professionals deem as a standard for test management and defect tracking. When anyone is evaluating such a tool, they want to compare it with QC, since it efectively combines both.
Although there doesn’t seem to be a single popular open source alternative to QC, there are diferent tools available for defect tracking that can be combined with test management tools to meet all QC features.
But one key issue is that such combinations bring considerable overheads with having to support and maintain multiple application systems. All these need to be weighed in before making a fnal decision on which tool is right.
Source: www.testingexperience.com, Testing Experience – The Magazine for Professional Testers