WeekNight Testing Take 5

Introduction

I had a quick exchange of words with Sharath (facilitator) before the session suggesting I will be joining late and got the reply ‘don’t b too late...because we have a very interesting session’. It indeed was an interesting session where all of us were as enthusiastic as the facilitator to see everyone enjoy testing the applications. Unfortunately, due to personal difficulty, Sharath was not around for much of the debriefing period to see the climax of the session when Darren McMillan kindly stepped in and kept us well informed in his absence.

Participants

Sharath Byregowda (facilitator)Martin Jansson, vamshi, Lalitkumar S. Bhamare, Stephen Hill, David Liebreich, Oliver V., Mohinder Khosla, Chris Thomson Philip, Adam Yuret, Kavitha Deepak, Darren McMillan, Del Dewar,  Richard James Hill,  Sudhamshu Ailineni, Brian Osman

About Mission Charter

The session started in good humour, as ever, with everyone hoping to make the most from the session. The facilitator was sure that no one will be disappointed with the missions that followed and it would give everyone the opportunity to collaborate and discuss results openly.

Mission 1: An addictive fun cat game to Circle the Cat. This is a game that you can play by clicking green dots to stop the cat escaping the board.

Mission 2: Find bugs with Horse Dressage game

Briefing:

❝As testers, testing are one part of our job but we are also expected to investigate research and provide information to our stakeholders. Today the mission is to investigate, analyse and find a solution and yes find bugs on the way❞


❝If you meet the first mission within the hour move to the second mission ❞

These are two contrasting missions, the former expecting an outcome of a solution to solve it whereas the latter was more like finding what’s wrong with the application.

Thinking about Testing Approaches

Whenever I approach testing I ask myself the obvious

1.       Do I understand the mission?

2.       Do I have enough information to start the mission?

3.       Do I need to speak to the customer, developer or BA to clarify requirements?

4.       Do I need any tools to record the test session, are these tools flexible enough to manipulate the test output meaning results for presentation?

5.       Does the application resemble anything like I have tested before?

There are two other things that I consider as part of test strategy to mitigate risks.

• Create a Persona(s)

Whether I am doing manual scripted testing or exploratory I make a mental picture of the user who is going to be using the application that I am testing. I speak to the user to build his profile and establish

·         what kind of user he is

·         what role he plays

·         what level of experience he has with similar applications

·         what are his pain points

Based on this information I draw up a mental sketch of the user that I impose on the tests I plan to do. If I am doing scripted testing then in the title section of the test I record the name that I have given to the persona. If tests are written in an automated tool or Excel then each test is prefixed with the persona name. So when it comes to analysing the results I can run a query to bring the results in a report to show the status to the manager or the user for a particular persona. In the case of exploratory testing I do the same tag each test with the persona during the testing session. In the current mission I gave the name ‘gamer’.

Personas are normally created at the start of a project to tag each story with the user during the story mapping stage. Similar approach is applied when writing acceptance tests to help build live documentation that Gojko Adzic advocates in his coming up book on Specification by Example.  I have been using personas for many years without mentioning the word persona to anyone till now. I am pretty sure if you start using it you can see the power behind the simple concept.

• Think about Heuristics

This is also very important aspect of exploratory testing to identify the heuristics that may be applicable in a particular situation. I tour to build a model of the product before deciding which heuristics would be appropriate that forms part of the tests.

When deciding a heuristic for mission 1, I was thinking of using map making heuristic but it turned out that I was actually applying trial by error heuristic instead.

In the case of mission 2, the choice was pretty much simple for me given the features of the game that follow a sequential order. I chose to use OFAT-one factor at a time ie changing the colour of the horse and clicking each menu option at a time.

Stakeholder Collaboration

One of the principles of Agile Manifesto is “Customer collaboration over contract negotiation”. I believe that success of any project depends on the active involvement of its stakeholders during each stage of the project. In our case we felt that such interactions with the customer were missing to some extent although there were plenty of co-operation from other testers on the mission which is a good thing. We had a number of pertinent questions that were not addressed as we progressed through the session. In order to meet the objectives of the mission to investigate, analyse and present information to the stakeholder, we felt that stakeholders co-operation is necessary if we are to be successful complete the mission.

In such situations, at least we should have been told that stakeholders would answer all queries related to the mission during the investigation stage if available. We felt that the information from the stakeholder was not forthcoming when it was needed that resulted in us making assumptions by looking back at our own experiences from past projects or agreeing among ourselves what should be the right answer in such situations. This, in my opinion, is not way the investigations in real projects are conducted. Full co-operation of the customers is vital in reaching right solutions through investigations.

Questions Asked Assumptions Made

In any projects there are bound to be questions from testers before, during and at the completion of testing phase. Questions such as

                Who is our stakeholder?

              What is the scope of testing?

            How the results will be represented and reported and to whom?

          How the results would be interpreted and used for?

        What information should be included from the investigation?

       Which of the missions would add high value and has high priority?

      Whether it is more important to just collect information or 
     timely information?

Would you assume that our mission facilitator is our stakeholder unless it is conveyed to you? I don’t think so. The mission was to ‘circle the cat’ and to investigate, analyse and present information about the product. We were desperate to find out whether we should complete mission 1 before moving on to the second mission or carry out deep investigative analysis for mission 1. We assumed that it was mission 1 objective to figure out patterns for the product and come up with a simpler solution. I didn’t think that was the right decision but we did not have much to go on.

Mission results

The key aim of the first mission was to encircle the cat but it was not clear what information is to be presented to non-existent or so called obtuse stakeholder. The mission statement expected information from the investigation and analysis of results with no idea how it should look like and what the content should be. This left us pondering when to finish with mission 1 and move onto the second. We felt whether we have done enough justification to mission 1 before continuing to next mission. The question came up whether spending too much time on one mission would jeopardise the second mission. In reality there were no bugs to be found from the first game so we were keen to jump straight into second testing challenge in order to find out whether it would be more fun and better use of our testing skills to report bugs.   

Circle the Cat

All of us approached circling the cat in different ways. To lighten up the atmosphere one of the participant even suggested photocopying the game board and drawing a circle around the cat with a pen. We were looking for a pattern to emerge but could not find one. There was consensus on the approach that we were all taking ie if the reset key is clicked multiple times to bring up at least 10 or more green dots on the board before attempting to circle the cat, we have a better chance of success. After couple of tries, using trial and error, I came to the conclusion that if you have the dots evenly scattered over the board except one corner have more dots than the rest you could corner the cat to the densely populated dots to trap it much easily. Others tried to draw a wider circle to achieve the same result. As one says there is more than one way to skin a cat and it turned out to be true as shown by the images below. You can watch the full sequence here. One of the sequences is shown here. If you want to follow up the discussions about the game then use the link.

3 Patterns of Circle a Cat

Horse Dressage

We needed clarity about this mission whether the goal is to find just bugs and not to report design or other information to the stakeholder. In spite of this confusion we carried on regardless. It was also not obvious how we should report bugs that we find. In other weekend testing sessions we are advised to use tools and techniques at our disposal and sometimes there is a mechanism to log bugs as well. I believe, for clarity or completeness, we need a similar system for week night testing to all tracked bugs and issues that are reported. This would help in providing feedback to a potential product owner if it is part of the brief.

Although, we discovered some bugs with the application but were not able to log them and present them in the debriefing for discussion. We concluded that the application is so buggy that it failed QA testing and should be returned to the developer. The captured video recording is shown here.

Domain Knowledge Matters

During the course of testing, we experienced that some domain knowledge would have helped.  This certainly would have helped construct better tests rather than firing random tests to explore the application. For example in the case of horse dressage(see the definition below), horse person would have spotted that there is no collected/extended canter offered, therefore, would have connected it to missing features, design fault or even inaccessible features. The other interesting thing that a horse person would have picked up about the movement of horse that it should be linear then walk, walk before trot and trot before canter. This shows domain knowledge has added value.

❝ the dressage is occasionally referred as ‘Horse Ballet’ whose purpose is to develop a horse's natural athletic ability and willingness to perform. ❞

It is a flash game that displays some of the dressage routines. If you are not used to testing such games then you might initially struggle before getting used to but it is not considered a handicap.

Message Got Lost in Translation

We were making all sorts of assumptions and draw conclusions of our own whether one should find out the best patterns to the game or a single solution that we all agree. Some of us thought the mission 1 is a puzzle while others said it a game. The message to complete mission 1 before moving on to mission 2 was not conveyed properly. There was nothing in the mission to identify patterns and understand how you circle the cat. Although mission clearly stated that it is to ‘circle the cat’ but still some were calling it trap the cat. These two contexts might sound similar but they are not. The dictionary meaning of circle is ‘A plane curve everywhere equidistant from a given fixed point, the center’ whereas trap means ‘to prevent from escaping’. I wonder how many of us exactly circle the cat by definition because I saw no evidence of that. 

Concluding

Both weekend and weeknight testing provide great opportunities for testers around the world to work together and share their experiences through the medium of social media. Testes working together pick up excellent tips from each other which they can apply to their own workplace. It not only gives testers exposure to solving problem skills that they would not have otherwise had. Some of the lessons learnt from the simple problem solving exercises above and discussing the observations openly with others give you more confidence that is crucial to your self-learning. You feel you are among friends who can guide you through issues that may help you in your future testing assignments. By working with guys whom I have not met boosted my moral to go out and explore new territories which I was reluctant to do before.  

My thanks to all participants who came up with innovative questions and made constructive contributions that made this report possible. Please feel free to comment for the benefit of others and add bits that I missed and you feel strongly should have been included in this report.