WTANZ#12: Boundary Testing: Sunday 30 January 2011, 630pm NZT
This was the first for 2011, for the Australia/NZ region. The participants were Richard Robinson as facilitator, Marlena Compton(AUS), Michael Larsen(WT Americas) and later Vamshi(Ind).
MISSION: do a boundary analysis on the parking calculator and find the maximum fee the calculator will give you.
The website address is: http://www.grr.org/ParkCalc.php
You have 30mins to complete this, at this time everyone will have a chance to present their boundaries, and feedback. You may raise or note down any assumptions you might make.
The participants went straight to work identifying that the calculator did not offer much in the way of errors for invalid or ‘ridiculous’ data ranges.
We were seeing parking fees of over $1,000,000 for periods before Christ to well after when the Terminator came from.
The participants did well to uncover many boundaries for each field in the calculator form. They tried numbers, mal-formed dates, incredible date ranges, non-sensical values, alpha. They threw the sink at the website form to test out their boundaries they had thought of.
Now, parking fees of over $17,000,000 were being discovered, and climbing. Negative dates, nulls, holding down keys, zero intervals, negative intervals, 35 month years, gregorian calendar dates, special charactesr, all trying to elicit error messages that were ever-absent.
Frustration was creeping in. The calculator wouldnt not break, it kept giving answers!
New information was found on a related web page which showed the parking rates and how they are enforced. This helped to find new boundaries, and explained why $2 was being charged when a parking time interval was zero.
Michael raised the topic of ‘rational boundaries’ and we discussed assumptions around what we define as rational and how we should handle that.
We started to hit the big numbers now… and URL values were being analysed for manipulation and boundaries.
3.39800571798E+13 Days, 22 Hours, 53.8666666667 Minutes
Overall, the session went very well with the participants identifying many boundaries and some very large parking fees.
It was good to see that they both actually used the calculator to identify and confirm boundaries which is a lesson to all of us. We are testers, and therefore should be ‘testing’ as much as possible. Rather than reading a spec and writing test scripts only. It is essential that the product is played with to perform robust analysis, wouldn’t you say?
Both testers started off using common sense values, then gradually got further and further from sense as they tried everything they could imagine. What developer would ever think to do their unit tests in this way? Many?
One type of input that was not identify was scientific notation. This would have assisted them to achieve the highest fee possible.
The session was a very enthusiastic and interesting excercise and the following themes were evident:
1. Assumptions: how do we handle them when we are analysing boundaries?
2. Available info: how much notice do we take of the boundaries spelled out to us?
3. Reporting: should we makes notes as we go, or just do this from memory at the end of the session?
4. Nature of boundaries: do we list the fields with sample data? data type descriptions? or use complex notation?
We concluded the session by discussing the nature of scripted testing vs exploratory. And agreed that exploratory testing at the outset engages your brain and thinking. This leads to a much deeper level of testing and the surfacing or many more bugs. It avoids the pesticide paradox. We agreed that scripted testing is best used for smoke testing and largely regression testing. To rely on scripted testing stops thinking, limits the ability to find bugs, and sends a tester to sleep.
If anyone would like to do this exercise with me, catch me on Skype (richrob79), I am happy to take you through it and share my knowledge and experiences. I have had the pleasure of performing this exercise with James Bach, and also receiving feedback from him. I feel it my obligation to pass this experience on to others. I will post this on my blog in the near future (richrichrichrichrich.com)
Thanks to all participants of WTANZ#12
The highest fee found was not too far from the record, which stands at:
2.13503982335E+14 Days, 6 Hours, 49.6 Minutes
This was discovered and fine-tuned my Jon and James Bach and used a specific type of syntax notation 😉