TheAccessPond.com

If you are a software tester or a Software Quality Assurance (SQA) person then you have no doubt written bug reports. The reports, which are also called defect reports, are the medium used to transfer your knowledge about the bug to those who will be tasked with fixing it. This report will first go to those who make the decision to fix or not fix the problem, the ‘Triage Team’.

It has been my experience that the subject of bug reporting is not covered properly in the schools and text books that deal with software testing. Even many of the biggest software development companies overlook the value of complete and effective bug reporting.

I gave a lecture on this subject at the California State University Northridge (CSUN) Technology & Persons with Disabilities Conference back in March of 2008. Since then I have seen signs that this trend is starting to change. The feature article in the September 2008 issue of “Better Software” magazine would be one such sign.

Now I am not sure if the person who wrote this article was at my lecture or not, but I can say that I am glad to see that bug reporting is starting to get some attention! The article mentioned above does a great job of going into detail on this. What I want to offer here is my approach to writing complete and effective bug/defect reports as it is more simplified and therefore easier to implement. I am not saying my way is best by any means. All I can say is that I have used it in the past and the method has worked everywhere I have used it.

While writing complete and effective bug/defect reports are important for all aspects of software development, it is even more important when it comes to issues that involved accessibility. Why? Because most people (Developers, Program Managers, Testers/SQA) are new to accessibility and often times don’t see the need to focus much attention on that subject. So the way in which you write your reports is going to have a bigger impact on changing this viewpoint.

What are the potential benefits of writing complete and effective bug reports?

• Time Savings for You
• Quicker Bug Fix Turnaround
• Tester Credibility

How is there a time savings for you? The more information you can provide the less likely you will be interrupted in the future with questions from the Program Managers and Developers looking further clarification.

You will also save time when the fix for that issue comes back to be verified. In this case you will spend less time trying to “remember” what the issue was and can jump right into testing the fix.

Don’t underestimate just how much time this can and will save you in the future!

How do you get a quicker turnaround on fixes? Because you are writing complete and effective reports these bugs will be triaged quicker and given to the development teams sooner.

You will build your credibility as a tester because as you consistently write complete and effective bug reports those who read those reports will start to notice the difference between your bugs and those of your peers.

As your credibility grows you may also experience less push back on your bugs because you will have gained the reputation of providing informative and accurate reports. If you are in an organization that considers the amount and quality of your bugs when the time for salary increase comes around, then you certainly don’t want to miss a chance to set yourself apart from the pack!

So there are benefits, I have personally experience those. You are likely wondering what to include in a bug report. Well I will tell you…

What to Include in a Bug Report

The format I follow is:

• Title
• Description
• Customer Impact
• Work Around
• Details
• Repro Steps
• Results
• Expected

Let’s take a more detailed look at each one of those bullet points.

Title
The title of your bug should be short but meaningful. What you want to accomplish here is to define the bug so that anyone looking at the title can understand the basic issue. This is important because when you go looking for this bug in your bug reporting system, you want to be able to distinguish it without having to open every bug to make sure you have the correct one.

For example an accessibility bug title could look like this:
“Project A Main page is missing skip navigation link”

Which is much more meaningful than this:
“Skip navigation link missing”

Another trick that is useful is to preface your titles with an identifier. This is not always necessary, but if you are tracking a lot of bugs and not all are dealing with accessibility then this can save you time by giving you a means to quickly sort your bugs by title or at least identify that the bug is an accessibility issue. Taking our example above you could create a title like:

“SEC508: Project A Main page is missing skip navigation link”
or
“ACC: Project A Main page is missing skip navigation link”

Choose any variation you like as long as it makes sense to you and those who will be reviewing your bugs.

Description
The description is a brief non-technical overview of the issue. The goal here is to convey the high level meaning of the bug so that those who are not the most technical people (i.e. managers) can understand what the issue is without having to come to you for clarification. Again, our goal is to save time by avoiding these interruptions

If done right, a manager will be able to know what the issue is based on the title description fields. They need not look any further.

Customer Impact
Many times bugs are deferred or closed as “won’t fix” because there doesn’t seem to be any clear customer impact. As a tester it is your job to put yourself in the customer’s shoes. If you can provide a details on how this will impact the customer negatively then outline that in the report.

What this does is show that you have given thought to the issue and what it will mean to the customer. This is something a lot of testers do not do! Talk about setting yourself apart from the rest of the testers!

Keep in mind that if there is no real negative impact on the customer we don’t want to make one up. You are building your credibility as a tester; don’t screw that up by stacking the deck with incorrect statements.

Work-Around
A bug may not be fixed if there is a valid work-around that is easily discoverable by a user. If such a work-around exists, call it out in the bug. Again, we want to state the facts even if it means our bugs may not be fixed. This all goes back to building our credibility as a tester.

Details
The details section is our catch all. Most importantly, this is where the technical details of the bug go. So what type of things may that be?

Call out what version of the OS and other software that may be involved in this bug. If you are using an Assistive Technology (AT) such as JAWS be sure to specify that version as well.

In addition to calling out the versions, if there is a tool being used, be sure to reference where a person can get a copy of that test tool. Also include the steps needed to execute the tool to run the test that discovered the bug. What this does is allow someone else to reproduce the bug and provides you a great summary of the steps you took initially. Again saving you time in the future when you need to verify the fix for the bug.

When you do provide instructions on how to use a certain testing tool or an AT that you used for testing, give the developer some insight on how they can also use the tool to verify their code. This means that the proactive developer will catch initial problems before they ever get to you. Again, saving you time!

If there is a certain accessibility requirement that has failed, you can also call that out in the details section. This saves time by identifying the requirement that has not been met. In the case of Section 508 requirements this means those looking at your bug don’t have to spend time trying to figure out which requirements have not been met. This can potentially save your time and certainly save the triage team’s time. Did I mention tester credibility?

For those testers who are a bit code savvy and take the time to dig in to the code to see what the issue might be, include the code snippet in the details section. Any developer worth his salt will welcome the insight. Even if you are wrong in your debugging efforts it still gives the developer a good place to start and I haven’t met many who frown on getting a little help.

You have carte blanch when it comes to the detail section. Just remember to keep the information there relevant to the bug and professional so that the reader is not wasting time. The idea is to provide all the necessary information that will allow the triage team and developer to address the issue without having to come back to your for clarification.

The article mentioned above has some good insight and advice on the amount of content to include. Check it out!

Repro Steps
The repro steps section is where you spell out, step by step, what you need to do to reproduce the bug. Don’t assume that the person attempting to do this will know how to use the test tool or how to navigate to a particular dialog etc. Take the time to include everything here,if only for the sake of documenting it for your own needs. Too many times in the past I have had a bug returned to me for verification 6 months, 9 months or a year after I wrote the bug. Having these steps clearly documented saved a lot of time!

Results
The results section is where you outline what happened. It is pretty simple and straight forward, no?

Expected
Under the expected results you stated what should have happened. Why is this important? Because sometimes what you expect to happen and what the developer expected to happen can be two different things. In situations like this you may find a design change happened because of your bug! A feather in your cap!

The other benefit of knowing what to expect is that you give the pass criteria for anyone else who may need to create the fix or verify the fix for this issue. Again, tester credibility comes to mind.

In summary, this post is in no way meant to be a complete or comprehensive list. Many pages could be written on this subject and each person may have a different view as to what should and should not be included. I can tell you that this method has worked for me. I offer it to you as a way to determine what will work best for you and to hopefully get you thinking about the quality of the bug/defect reports you create. With that in mind I have included a sample report below that follows the guidelines outlined above and a Word doc that you can download and use a template as you see fit.

All I ask is that if you find success with this method of bug reporting, that you drop me a line and let me know.

Sample Bug Report

Bug Reporting Template

No Comments »