Quality is not only an output; it’s an outcome over the output

You may think what the differences between these two are, they sound synonymous. It’s true that they are, but there are some differences.

Let’s first talk with the definition of these twos.  In my sense, in the non-profit world, outputs are programs, training, and workshops; outcomes are knowledge transferred and behaviors changed. In the material world, the distinctions are not always so clear. Let's define outputs as the stuff we produce, it may be physical or virtual, for a specific type of customer—say, Foot over-bridge for the people. And let's define outcomes as the difference that it makes—keeping the people safe from the road accidents.

Outcomes are the benefit your customers receive from your stuff. This starts with truly understanding your customers' needs—their challenges, issues, constraints, priorities—by walking in their shoes and in their neighborhoods, businesses, and cultures. See what's inconvenient, taking a lot of time, money, and/or effort.

Outputs are important products, services, profits, and revenues. Outcomes create meanings, relationships, and differences. Outputs, such as revenue and profit, enable us to fund outcomes; but without outcomes, there is no need for outputs. 

Considering these definitions and differences, let’s come to the world of software and services. Here, the product itself i.e. any software or service is an output. And, the needs that have been fulfilled by it are the outcomes. So, what the users or audiences look forward to it are their needs and wishes, not actually the product itself. So, if the product cannot fulfill their needs and wishes, it is not the outcome. It is only an output with no uses. And, also if the product can have the features to fulfill the needs, but somehow the users or audiences cannot use it for their needs, it is not the outcome as well. Here comes the use of the term ‘Usability’. If a product or service is not usable enough to get the work done although there is major fault in its functionality, then there is no outcome. So, usability is an important issue to let the user finding the outcomes. So, as QA or Test engineers, it will be our responsibility to look for fine-tuning the outcomes along with the output instead of looking for the output only.

Discovering Heuristics

At first, I would like to convey my special thanks to James Bach for the idea and inspiration to write this blog post about my own heuristics which I follow while testing. Actually, I was not aware of the heuristics that I follow unconsciously while testing; James helped me to find out these.

Initially, I would like to discuss briefly about Heuristics and then I will explain what approaches I usually follow in my own work.

Heuristic refers to experience-based techniques for problem solving, learning, and discovery that gives a solution which is not guaranteed to be optimal. Examples of this method include using a rule of thumb, an educated guess, an intuitive judgment, stereotyping, or common sense. [Wikipedia]

[A heuristic is a fallible method of solving a problem or making a decision : James Bach] According to James, Heuristics are not experience-based, they can be theory-based. 

Applying heuristics needs two things generally. First, we have to learn about heuristics. Being Knowledgeable about heuristics gives us the idea to apply and encourages us to develop our skills.  Second, we should understand when a heuristic or method is helpful, when it should be used and when it should not. Moreover, we should learn how to frame our own problem so we can apply the best heuristic to solve the problem.

To develop the heuristics, we have to continuously look for ideas in testing that frequently get us the results. Perhaps, some ideas that often help us to uncover the defects in our own way. For that, we have to look for the patterns or commonality that we follow consciously or unconsciously while testing a product.

Now, let’s talk about what I have found when I tried to find out my own heuristics:

Most of the time at the beginning of testing any product, what I do is without looking into any specification/requirements/document, I just look into the product with a blank mind to find out what is the product actually. In my sense, this helps me to remove any kind of biases which I might have.  Sometimes it is helpful and sometimes it is not. But, it helps me to look into any product with the fresh eyes. And, fresh eyes without any bias helps to generate so many questions and which results in broadening the ideas/scenarios for the particular product.

So, in my case, playing with the product without knowing about it may be a heuristic. May be we can term it as 'Ignorant Heuristic'.

Another testing strategy that I usually follow is, after testing a product/feature for a certain period of time , I do give a pause on the testing and go to other things (May be testing other product or anything that is not related to the product itself) to give my mind a break so that it can start thinking again from the beginning. Then, I again come back to testing of that product. (Here, this ‘Certain Time’ is totally dependent on the context and my mind. May be, I can come up with another heuristic which may tell how I can decide on this ‘Certain Time’)

Most of the times I have found that, while doing the other things in the break, some thoughts which were not connected while I was doing the test, get connected unconsciously in my mind and help me to find a better scenario. May be, that is how my mind works. It might need a break to connect the unconscious ideas together which I experienced during the testing. Most of the times, I found that, there are some simple ideas/concepts which I missed or couldn't grasp well at the time, but can reconnect them on the breaks. Sometimes, there are some critical ideas as well which I can reconnect at the breaks.

So, giving your mind a break may be a behavioral skill/heuristic to follow while testing something. In my case, it works. May be, we can term it as "Kit-Kat Heuristic" (From the tagline of Kit-Kat: Have a break, Have a Kit-Kat)

There may be many more heuristics like the above two for me and there will be a lot for you as well. The thing is, we need to find out these from our own way of doing things, so that we can use them wisely and effectively in our work.

[Courtesy: 1. Wikipedia
                 2. James Bach ]

The Art of Bug Reporting

                    "A problem well stated is a problem half solved." - Charles Kettering

After reporting a bug, if the developers come to you asking how to reproduce the bug, then you are actually being trolled. It might be for two reasons. One, the developers are not good at understanding and another, you are not communicating well through your bug reporting.

So, what should I include in my bug report? Well, that is a good question; I try to follow these:

1. Title: A short one line description where a programmer should be able to understand what type of bug is being reported.

This is the most important thing for a bug report. Because, usually developers/managers are not always interested to go through the details and they try to grab an idea by just reading the title. So, the title should be self-explanatory and must give a proper idea about the bug.

For Example, Bug Title can be as:

Module X: Clicking on the 'Print' link from ‘X’ page, redirects user to the Error page.

If your module name is ‘User’ and page name is ‘Managers’, then the title should be as follows:

User: Clicking on the 'Print' link from Managers page, redirects user to the Error page.

Here, anyone can guess the bug just reading the title.

 2. Description: Description is a more detailed paragraph which should be clear and informative, anyone who will read the description should be able to picture in his/her mind what is being described and imagine the bug scenario in his/her mind.

For Example, for the previous bug, the description can be as follows:

In the User Module, When a User clicks on the ‘Print’ link from the ‘Managers’ page, then user is being redirected to the Error page. Following error messages have been shown in the Error page:

The input is not a valid Base-64 string as it contains a non-base 64 character, more than two padding characters, or an illegal character among the padding characters.

This bug is occurring in every browser after the latest deployment on June 19,2013 in Test Server.

Check the attached image for more details.

3. Reproduction Steps:  This is a list of clear and easy to follow instructions, documenting every input and action you took to reproduce the bug. Most of the time, anyone can have the idea of the bug reading the title and description. But, for some complex and unusual scenarios,  Reproduction steps should also be added and it is very helpful in many cases.

For Example:

Steps to produce the bug:

1.  Login to User Module

2. Go to Members page

3.  Click on the ‘Print’ link

4. Check that the, Error page will be shown. Check the attached image.

4. Actual Results: Usually, a statement expressing what happened when following the above reproduction steps.

   Error page has been shown with following error messages….

5. Expected Results: This is usually one or two lines where the user writes what he/she expected when following the reproduction steps.

   In such scenario, User list should be printed out in paper(s).

6. Attachment: After writing the above into your report, additionally you can record a video and also take screenshots and attach these to the report. Screen capture software is available and some are free. Make sure to give a proper name for the attachment, not like ‘Untitled 1.jpg’ .

7. Environments: In which environment the bug actually happens. It may contain the Browser information, System information or any other information which covers the environment.

8. Severity: It is the extent to which the defect can affect the software or its end users. In other words, it defines the impact that a given defect has on the system or to the client.

For example:  If an application or web page crashes when a remote link is clicked, in this case clicking the remote link by an user is rare but the impact of  application crashing is severe. So the severity is high but priority is low.

9. Priority:  Priority defines the order in which we should resolve a defect. Should   we fix it now, or can it wait? This priority status is set by the tester to the developer mentioning the time frame to fix the defect.

For example: If the company name is misspelled in the home page of the website, then the priority is high.

10. Release Version: Sometimes there may be multiple code streams are being maintained for different release versions. So, mentioning the Release version may also help in this regard.

You have to keep in mind that, you may not grab these all points in a moment. But, by practicing day by day, you can be an effective bug reporter which will eventually lead you and your team to the success.  One important thing to remember that all these points may not be required in some cases or more items may be added for some cases. It actually depends on the context. Here, you have to be the master of understanding your own context.

A cartoon from the Cartoon Tester blog may give you a good idea about all the above points: