Date: 20th March 2010
Time: 15:30 – 17:30 GMT
Mission:
To evaluate the use of Ben Simo’s usability heuristic for error reporting. Select at least one error you have encountered recently, and use the FAILURE heuristic to evaluate it.
Submit a report evaluating the heuristic.
Descriptions of the heuristic are available here. If you have not had the opportunity to read the post before the session, then I’d recommend just reading the first one to get you started.
Aspects you might consider in the report should include (but are not limited to):
- Whether the heuristic was useful to you.
- Were there other items you could add to the heuristic?
- If there were other items you felt you could add, you may wish to create your own mnemonic for your new heuristic.
Testers: Phil Kirkham, Shruti Gudi, Ajay Balamurugadas, Anna Baik, Markus Gärtner
Anna Baik facilitated the discussion. Each tester was asked about their experiences.
Markus Gärtner started the discussion. He shared his experiences. He had encountered an error message in a game recently, which he needed to reproduce for the session. He noticed that they all applied in this regard. In addition he tried to apply the mnemonic to his application at work, for which some of the parts did not apply. Since he deals with an application within several subsystems, the UI for example was not applicable.
Ajay Balamurugadas continued the debrief. He stated that he achieved his personal mission of taking half an hour to attend the skype chat, and taking half an hour on the actual mission giving. The mission was easy from his perspective. There are some more questions he wanted to add to his list for the next time. Basically he took the whole list of questions and added answers in his report. Asked on the applicability of the heuristic, he stated that he uses it, since it’s very useful. He continued on his particular error at youtube. While uploading a new profile picture, the error message was misleading, so the heuristic applied there. Ajay pointed out that the questions Ben provided for the mnemonic where actually a sort of a checklist. Therefore they are very useful. For the addition, he explained that he wanted to add reproducibility to the heuristics.
Shruti continued. She explained that she evaluated an error she recently got on her mobile phone. She explained that from the mnemonic the logfile and the support contact did not apply in this domain. She noticed that the error looks different from the QA perspective as from the user’s perspective. In the discussion, the participants found out that these differing perspectives could be related to the emotions in the FAILURE heuristics. Shruti finished with stating that she will definitely use the mnemonic again. The UI and the impact part could be generalized more from her point of view. During the discussion the participants identified that taking a closer look on the users and the stakeholders of the application is a worthwhile thing. Thereby the user of the system could be also another software system. Audible and visible error feedback was also mentioned as a clearly helpful way to provide the user feedback on how to resolve the error by themselves.
The chat transcript from today’s session can be found here.
Pradeep Soundararajan is a well known tester in the Testing World. Pradeep is a Test Consultant, Coach, Speaker and Mentor to many testers across the world. He is also the Author of 
awesome, looks like it was a great discussion. i woke up late and missed it all. I will try login for the next session.
agreed, FAILURE heuristic is really helpful, i will apply this to my next encounter and see how it goes.
I have since refined the heuristic to simple statements instead of the more open ended questions. However, those questions, and more, are still part of my application of the mnemonic.
F unctional: The error detecting, reporting, and handling functions properly.
A ppropriate: Errors are reported at the appropriate time in an appropriate manner.
I mpact : The impact of the failure is communicated to those impacted.
L og: The error is appropriately logged; or not logged.
U ser Interface: The error is reported in the user interface in terms that the user understands.
R ecovery: The user is informed of what they need to do to recover from the error.
E motions: The error handling and reporting has a positive influence on user emotions.
I am so sorry I was unable to join this session. Here are a few quick comments on what I see above. I’ve not yet read the transcript.
Markus: Even a backend subsystem has a UI. The user might be another piece of software. So for the UI part, I’d consider if the reporting provides appropriate information to that other system so that it can detect and recover from the failure too. Also, I find that poor error reporting to human users is often the result of poor reporting in 3rd party components; and the developers of the system know no more about the error than they show to users.
And yes, not every part of the heuristic will apply in all situations.
Ajay: Add whatever questions you find useful. Mine were intended to be examples to help get people thinking. It seems you’ve used them as such.
Shruti: Yes! Error reporting looks different based on the readers perspective. Programmers and testers who understand a system’s architecture may find an error sufficient while it may only add to user frustration.