Quality is not only an output; it’s an outcome over the output

You may think what the differences between these two are, they sound synonymous. It’s true that they are, but there are some differences.

Let’s first talk with the definition of these twos.  In my sense, in the non-profit world, outputs are programs, training, and workshops; outcomes are knowledge transferred and behaviors changed. In the material world, the distinctions are not always so clear. Let's define outputs as the stuff we produce, it may be physical or virtual, for a specific type of customer—say, Foot over-bridge for the people. And let's define outcomes as the difference that it makes—keeping the people safe from the road accidents.

Outcomes are the benefit your customers receive from your stuff. This starts with truly understanding your customers' needs—their challenges, issues, constraints, priorities—by walking in their shoes and in their neighborhoods, businesses, and cultures. See what's inconvenient, taking a lot of time, money, and/or effort.

Outputs are important products, services, profits, and revenues. Outcomes create meanings, relationships, and differences. Outputs, such as revenue and profit, enable us to fund outcomes; but without outcomes, there is no need for outputs. 

Considering these definitions and differences, let’s come to the world of software and services. Here, the product itself i.e. any software or service is an output. And, the needs that have been fulfilled by it are the outcomes. So, what the users or audiences look forward to it are their needs and wishes, not actually the product itself. So, if the product cannot fulfill their needs and wishes, it is not the outcome. It is only an output with no uses. And, also if the product can have the features to fulfill the needs, but somehow the users or audiences cannot use it for their needs, it is not the outcome as well. Here comes the use of the term ‘Usability’. If a product or service is not usable enough to get the work done although there is major fault in its functionality, then there is no outcome. So, usability is an important issue to let the user finding the outcomes. So, as QA or Test engineers, it will be our responsibility to look for fine-tuning the outcomes along with the output instead of looking for the output only.

Discovering Heuristics

At first, I would like to convey my special thanks to James Bach for the idea and inspiration to write this blog post about my own heuristics which I follow while testing. Actually, I was not aware of the heuristics that I follow unconsciously while testing; James helped me to find out these.

Initially, I would like to discuss briefly about Heuristics and then I will explain what approaches I usually follow in my own work.

Heuristic refers to experience-based techniques for problem solving, learning, and discovery that gives a solution which is not guaranteed to be optimal. Examples of this method include using a rule of thumb, an educated guess, an intuitive judgment, stereotyping, or common sense. [Wikipedia]

[A heuristic is a fallible method of solving a problem or making a decision : James Bach] According to James, Heuristics are not experience-based, they can be theory-based. 

Applying heuristics needs two things generally. First, we have to learn about heuristics. Being Knowledgeable about heuristics gives us the idea to apply and encourages us to develop our skills.  Second, we should understand when a heuristic or method is helpful, when it should be used and when it should not. Moreover, we should learn how to frame our own problem so we can apply the best heuristic to solve the problem.

To develop the heuristics, we have to continuously look for ideas in testing that frequently get us the results. Perhaps, some ideas that often help us to uncover the defects in our own way. For that, we have to look for the patterns or commonality that we follow consciously or unconsciously while testing a product.

Now, let’s talk about what I have found when I tried to find out my own heuristics:

Most of the time at the beginning of testing any product, what I do is without looking into any specification/requirements/document, I just look into the product with a blank mind to find out what is the product actually. In my sense, this helps me to remove any kind of biases which I might have.  Sometimes it is helpful and sometimes it is not. But, it helps me to look into any product with the fresh eyes. And, fresh eyes without any bias helps to generate so many questions and which results in broadening the ideas/scenarios for the particular product.

So, in my case, playing with the product without knowing about it may be a heuristic. May be we can term it as 'Ignorant Heuristic'.

Another testing strategy that I usually follow is, after testing a product/feature for a certain period of time , I do give a pause on the testing and go to other things (May be testing other product or anything that is not related to the product itself) to give my mind a break so that it can start thinking again from the beginning. Then, I again come back to testing of that product. (Here, this ‘Certain Time’ is totally dependent on the context and my mind. May be, I can come up with another heuristic which may tell how I can decide on this ‘Certain Time’)

Most of the times I have found that, while doing the other things in the break, some thoughts which were not connected while I was doing the test, get connected unconsciously in my mind and help me to find a better scenario. May be, that is how my mind works. It might need a break to connect the unconscious ideas together which I experienced during the testing. Most of the times, I found that, there are some simple ideas/concepts which I missed or couldn't grasp well at the time, but can reconnect them on the breaks. Sometimes, there are some critical ideas as well which I can reconnect at the breaks.

So, giving your mind a break may be a behavioral skill/heuristic to follow while testing something. In my case, it works. May be, we can term it as "Kit-Kat Heuristic" (From the tagline of Kit-Kat: Have a break, Have a Kit-Kat)

There may be many more heuristics like the above two for me and there will be a lot for you as well. The thing is, we need to find out these from our own way of doing things, so that we can use them wisely and effectively in our work.

[Courtesy: 1. Wikipedia
                 2. James Bach ]

The Art of Bug Reporting

                    "A problem well stated is a problem half solved." - Charles Kettering

After reporting a bug, if the developers come to you asking how to reproduce the bug, then you are actually being trolled. It might be for two reasons. One, the developers are not good at understanding and another, you are not communicating well through your bug reporting.

So, what should I include in my bug report? Well, that is a good question; I try to follow these:

1. Title: A short one line description where a programmer should be able to understand what type of bug is being reported.

This is the most important thing for a bug report. Because, usually developers/managers are not always interested to go through the details and they try to grab an idea by just reading the title. So, the title should be self-explanatory and must give a proper idea about the bug.

For Example, Bug Title can be as:

Module X: Clicking on the 'Print' link from ‘X’ page, redirects user to the Error page.

If your module name is ‘User’ and page name is ‘Managers’, then the title should be as follows:

User: Clicking on the 'Print' link from Managers page, redirects user to the Error page.

Here, anyone can guess the bug just reading the title.

 2. Description: Description is a more detailed paragraph which should be clear and informative, anyone who will read the description should be able to picture in his/her mind what is being described and imagine the bug scenario in his/her mind.

For Example, for the previous bug, the description can be as follows:

In the User Module, When a User clicks on the ‘Print’ link from the ‘Managers’ page, then user is being redirected to the Error page. Following error messages have been shown in the Error page:

The input is not a valid Base-64 string as it contains a non-base 64 character, more than two padding characters, or an illegal character among the padding characters.

This bug is occurring in every browser after the latest deployment on June 19,2013 in Test Server.

Check the attached image for more details.

3. Reproduction Steps:  This is a list of clear and easy to follow instructions, documenting every input and action you took to reproduce the bug. Most of the time, anyone can have the idea of the bug reading the title and description. But, for some complex and unusual scenarios,  Reproduction steps should also be added and it is very helpful in many cases.

For Example:

Steps to produce the bug:

1.  Login to User Module

2. Go to Members page

3.  Click on the ‘Print’ link

4. Check that the, Error page will be shown. Check the attached image.

4. Actual Results: Usually, a statement expressing what happened when following the above reproduction steps.

   Error page has been shown with following error messages….

5. Expected Results: This is usually one or two lines where the user writes what he/she expected when following the reproduction steps.

   In such scenario, User list should be printed out in paper(s).

6. Attachment: After writing the above into your report, additionally you can record a video and also take screenshots and attach these to the report. Screen capture software is available and some are free. Make sure to give a proper name for the attachment, not like ‘Untitled 1.jpg’ .

7. Environments: In which environment the bug actually happens. It may contain the Browser information, System information or any other information which covers the environment.

8. Severity: It is the extent to which the defect can affect the software or its end users. In other words, it defines the impact that a given defect has on the system or to the client.

For example:  If an application or web page crashes when a remote link is clicked, in this case clicking the remote link by an user is rare but the impact of  application crashing is severe. So the severity is high but priority is low.

9. Priority:  Priority defines the order in which we should resolve a defect. Should   we fix it now, or can it wait? This priority status is set by the tester to the developer mentioning the time frame to fix the defect.

For example: If the company name is misspelled in the home page of the website, then the priority is high.

10. Release Version: Sometimes there may be multiple code streams are being maintained for different release versions. So, mentioning the Release version may also help in this regard.

You have to keep in mind that, you may not grab these all points in a moment. But, by practicing day by day, you can be an effective bug reporter which will eventually lead you and your team to the success.  One important thing to remember that all these points may not be required in some cases or more items may be added for some cases. It actually depends on the context. Here, you have to be the master of understanding your own context.

A cartoon from the Cartoon Tester blog may give you a good idea about all the above points:

Testing the developers, along with the development…

Interesting topic, right? What do you think ?

If you think this is interesting, then lets talk about this a bit more. 

Moving on...

Whenever we are testing a product or development, it is definitely from a development team. As we have been working with this team from long times, we must have some ideas and knowledge about the developers. So, can this awareness affect our testing effort or shall i say, can this awareness benefit us or not?  Lets share some experiences on it.

Knowing the developers or what I can say testing the developers can be advantageous or can be harmful. Lets talk about the benefit and negatives of this by examples.

Lets assume a 'Bug' to be a 'Criticism' (though, a bug is a blessing for me and for all the testers as well).  As a human being, we always do one common mistake. That is called 'Stereotyping'. So, whenever we are being given a new issue or a new feature from a specific developer (Say Developer X), unintentionally sometimes we take it as granted that Developer X tends to make some common mistakes like not thinking of the negative scenarios, spelling mistakes. As a result, we do strict our thinking ability and we loss our enthusiasm to explore for the unwanted things. This is a dangerous thing which affects directly on the curious mind and let it stop to work profoundly.

On the other hand, sometimes it benefits us while we are in a rush to test a some specific issues before release of the product. When you are in hurry what we usually do is finding out the minimal ways to complete the tasks in hand. So, we need to focus on the particular important things. So, if you do some tests focusing on the important parts and keeping in mind that Developer X has done it, then you can find out the most important features which will not be left out for testing before the release or when you are in a hurry.

 I know all the specialists of Testing whom i really follow, will not support this. Actually, i do not like it as well. But, sometimes it comes up through the unconscious mind and i have tried these approach unconsciously. Well, i had some success as well. But, in the long run, it actually limits the broader thinking approach.

So, testers!!! Be aware of this 'Stereotyping' !

Selective Retention while Testing

We have so many organs in our body and if we are lucky enough then we have these receptive organs with full of functionality. So, in every moment we are having lots of inputs through these. But, as we have the most obscure yet most creative thing in out head that is 'Brain', we do always try to conserve those inputs which seem important to us and leave the others. Trust me, we are continuously doing it in every moment either consciously or unconsciously. So, this is what the topic is all about. This is called Selective Retention.

Formal definition of Selective Retention from Wikipedia:

"Selective retention, in relating to the mind, is the process when people more accurately remember messages that are closer to their interests, values and beliefs, than those that are in contrast with their values and beliefs, selecting what to keep in the memory, narrowing the informational flow."

According to an article from Illinois State University, Selective Retention is the personal filtering of what we see and hear so as to suit our own needs. ISU also says, Much of this process is psychological and often unconscious.

Well, now some questions may come to your mind like "What is the uses of it while testing?",  "Testing should cover every inputs in all dimensions, so are you kidding about testing?". For those who do not agree with me, please be with me for the rest of the post. I will try to explain my thoughts regarding this. If you will not agree after completing the post, then i will definitely say that i am a failure to justify my reasoning. But, i wont say that 'Selective Retention is not important'.

Now, come to the core area of our expertise that is testing. While testing, actually the term should be 'Exploratory Testing', you are the king of the domain. You can traverse anywhere you want and you can do and see anything you want to. So, one thing about Exploratory Testing is that you have no boundary condition. So, should we explore all the things? 
Theoretically, we should. But, we are not living in the world without limitations. Time and cost are the main limitations we are living with. So, we need to traverse and explore the domain with  a mindset that the important things should be completed effectively. Those who believe in Exploratory Testing will definitely agree with me in this point. So, here while testing we do always accept the information what is needed and try to retain them in our mindset. These practices become a habit while you are growing up as a tester. So, when you have grown up enough, there are always some selective retention in your way of testing unconsciously. It is not bad and it should happen because you are the king of your product and you need to act fast and spontaneously. So, if this is not unconscious, it can never be such fast as it should be.

This is all about the Selective Retention while Testing. So, are you guys are with me? I will appreciate your opinions, thoughts and whatever you want to say about it. 

Thank You !!!

Acknowledgement: 1.  http://en.wikipedia.org/wiki/Selective_retention
                              2.  http://www.citeman.com/696-selective-attention-distortion-and-retention.html

Load Testing of Database Query with Jmeter

Most of people who are involved in the testing activities have some ideas about Jmeter. JMeter is a popular open source tool for load testing, with many useful modeling features such as thread group, timer, and HTTP sampler elements. Interesting thing is that along with the usual load testing of a web application or a web server, Jmeter has the feature to test the load of the database. In this post i have tried to give a step-by-step procedure to test a simple database queries with Jmeter. I have a plan to make several blog posts to cover the important load testing scenarios of database. Here, i have done the load test for Microsoft SQL Server 2008. I have used the jTDS Driver (from net.sourceforge.jtds.jdbc.Driver class) to integrate the Jmeter with SQL server. You can test for the other databases (Oracle, MsSql, PGSQL ) also.

Required Software:

•  Jmeter:  http://jmeter.apache.org
•  Microsoft SQL Server (jTDS Driver): http://sourceforge.net/projects/jtds/files/

Steps to follow:

Step 1: Adding Thread Group Control

• Thread Group allows to run script with n^th no. of users with defined ramp-up period and also the scheduling of execution.
•  Right click on the Test Plan and add the Thread Group control.

                                                   Figure 1: Adding Thread Group Control

         

                                              Figure 2: Adding Thread Group Control

Step 2: Adding JDBC Connection Configuration control

•   Used to configure the database i.e. Oracle, MS SQL Server etc.
• Add JDBC Connection Configuration control to the Test Plan.

Database Connection Configuration

•         Database URL:  jdbc:jtds:sqlserver:// [server name or ip] / [db name]

•         JDBC Driver Class: net.sourceforge.jtds.jdbc.Driver

•         Username: Username of the database (e.g. Username: scott)

•         Password: Password of the user (e.g. Password: tiger)

Important:

Add the following Jars in JMeter Installation directory (i.e. D:\jakarta-jmeter-2.3.1\lib)

·         jtds-1.2.5.jar

          

                                         Figure 3: Adding JDBC Connection Configuration control

Step 3:  Adding JDBC Request Control

•   JDBC Request control allows to run a SQL query, procedure etc.
• Add the JDBC Request control below the Thread Group.

                                    Figure 4: Adding JDBC Request Control


In my db, there is a single table ("Student") which contains only two columns(id, name) and this table has only 2 rows. You can test for the 'Insert', 'Update' and 'Delete' commands also. For that, change the 'Query type' to 'Update Statement' from the drop-down box.

Step 4:  Adding Listeners: View Result Tree control

• View Result Tree control is used to view the result of the query executed.
•  Add View Result Tree control to the Test Plan.

                                  Figure 5: Adding Listeners: View Result Tree control

Step 5: Verify Result from the Listeners 


 Check the load testing results from the Listeners. 

                                        Figure 6: Adding Listeners: View Result Tree control

                                    Figure 7: Verify Result: View Result Tree control






In the next few blog posts, i will try to extend this load test with the stored procedures and reading data from configuration files etc.

Be 'Stupid' to be a 'Smart Tester'...!!!

"If we did'nt have stupid thoughts
we'd have no interesting thoughts at all"

Interesting, Interesting and say it loud that it is interesting. Lets first share a "Stupid" video with you all:

     

So, have you enjoyed the video? I think you must have enjoyed. If you haven't, then you are not that 'STUPID'....:P

So, what makes me share this post with you.I must have something in my mind surely about 'Testing' or shall i say 'Stupid Testing'...:)

When i saw this video yesterday on the Diesel website, i was truly amazed. Then, i thought about it for a moment and realized that 'Stupidity' is a part of my regular activity too. And, that is none other than 'Testing'. Its must be a part of all the other Testers' activity also.

Everyday for testing different modules, we have to do basically all the 'Stupid' things along with the 'Smart' things. It may be a 'Stupid' inputs in the text-fields or it may be a 'Stupid' flow of activities to find a bug that cannot be found if we do only the 'Smart' ('Valid'/'Regular') way of doing that.

Truly, when a developers commit a code, he/she must have had some positive tests('Smart') before passes it to the QA team. So, as a tester if we only do the 'Smart' positive tests, then no bug will be found literally. So, along with the 'Smart' positive tests, we have to incorporate some 'Stupid' negative tests to find out the loopholes of that committed code.

Honestly, as a tester we have to take the risk of finding out the critical bugs by various 'Stupid' things continuously. So, i'm not afraid of being a 'Stupid' tester. Because, more or less you must be a 'Stupid' tester to be a successful 'Smart' tester.

Last part of the lyrics that i've enjoyed from this video is given below:

Its not smart to take risks...
Its stupid.

To be stupid
is to be brave

The stupid isnt afraid to fail.
The stupid know there are worse things than failure...
like not even trying.

Smart had one good idea,
and that idea was stupid.

You cant outsmart stupid.
So dont even try.

Remember
only stupid can be truly
brilliant

So,
BE STUPID...:)


Courtesy: www.diesel.com