WTA-63: Test Strategy in the Context



Date: July 25th, 2015
Time: 1PM EST / 10AM PDT

Pre-Session Setup


Session Details

How change of context affects your test strategy? Or it doesn’t if functional requirements remain the same? How performing tests and learning about the application changes our understanding of context?
We’ll go through this exercise together and discuss the results.

Online Logistics

To participate in this WTA session, send us an email (WTAmericas AT gmail DOT com) and/or send a contact request to Skype ID “WeekendTestersAmericas”). Once you indicate you wish to participate, we will add your Skype ID to the session setup.

The day of the session, please contact us 20 minutes prior to the start of the session, so that we can add you to the group (if you email or Skype us in advance, we will add you based on that RSVP).

Facilitator: Albert Gareev

Participants: Michael Larsen, Damian Synadinos, Trisha Agarwal, Mark Halvorson, Saurabh

Experience Report

Despite of some comments I’ve seen today, in my pro career of a contracting testing lead the situation when functional requirements are given with little details about context occurred quite often. I also played a type of business person resembling what I observed and experienced. I just wasn’t mean and complaining about testers though that’s been typical. The particular “trap” I simulated here is that the business seems to think that testers only need functional requirements, and they can define a “QA Strategy” with estimates based on that.
The learning point for me as a facilitator was something that I didn’t even think about: the inertia. Even though the context #2 eliminated camera involvement, pictures were still considered. And “students” were still considered as users in the context #3.


Functional Requirements
The program receives three values. The three values represent the lengths of the sides of a triangle. The program recognizes whether the triangle is scalene, isosceles, or equilateral.
A scalene triangle is one where no two sides are equal.
An isosceles triangle has two equal sides.
An equilateral triangle has three sides of equal length.

A message from business.
We want a mobile app. User points camera at a triangle. The app takes measurements. Recognizes the type of the triangle.

Test Strategy and Risks
The major purpose of TriangleFinder is to help people identify types of triangles by taking pictures and determining the results.
Therefore, our primary concern in testing it is to evaluate the correctness of triangles that we photograph, and the ability of users to properly operate the product to obtain those decisions.
Although we will focus the bulk of our effort on those risk areas, we will also spend some time testing the general functionality of the
Our test strategy will consist of the following general test tasks:
– Understand the decision algorithm that determines the type of triangle that it is.
– Generate and photograph a large number of triangles to determine the accuracy of the product in determining what they are.
– Review the documentation, and the design of the user interface and functionality for sensitivity to user error.
– Test with scenarios that might compromise the validity of the presented results (camera angles, colors, background details, etc.)
– Using requirements documentation, user documentation, or by exploring the product, we will create an outline of product elements and use that to guide user-level capability and reliability testing of the product.

The principal issues in executing this test strategy are as follows:
– The difficulty of understanding and simulating the decision algorithm.
– The risk of coincidental failure of both the simulation and the product.
– The difficulty of automating decision tests.

Business context #2.
An important message from Business. An input from the testing team made us reconsider. Seems like it’s either too risky or too expensive to
develop such an app. This is a web app. User inputs values. Purpose: geometry study for students taking online courses and exams.

Test Strategy and Risks
– The camera was considered irrelevant with the change from “mobile” to “Web app”
– Testers request clarity on which browsers to support.
– Testing all versions of Internet Explorer was identified as a potential drag on resources.
– A new requirement was added: “The app gets the input and recognizes which triangle to help students study”
– A potential future requirement was identified: “we may get it integrated with the popular study platforms, if that matters. Maybe collect some user info for Business Intelligence.”
– Adding a BA role was brought up as a strategy to help clarify functional requirements and goals for testing.
– Testers requested a “mock up” of final product.
– Testers wanted clarification on what Automation tools would be appropriate, and what the ratio of Manual to Automated testing would be.
– Risks were identified as:  hardware , environment , and tools for testing and  maintenance of our tests as all our in different localization and testing
how effective we can collaborate and support to test effectively

Full session chat is uploaded.

About the Author

Albert Gareev is a dedicated testing and test automation professional, technical lead and a hands-on consultant. He combines passion for testing as investigation and learning with an automation approach helping to reach observation levels beyond "naked eye and hand" in complex, large-scale systems. Albert has a blog at http://automation-beyond.com/ and occasionally tweets at @AGareev