Charter Report from Richard Robinson: SaaS360

Weekend Testing: Saturday 31 March 2012, 1030pm, Sydney

Mission for today:

Register at Please do not test the registration page.

Once you are logged in, please focus on the following three areas:

* First Impressions.

* Features which are confusing.

* Garden paths - highlight the features/actions where the clicks can be reduced

I have chosen to do a charter for this because it meets my charter characteristics:

1. I have no idea what I am doing

2. I am to undertake simultaneous learning and planning

3. I want to capture all my early observations, because some of them might be useful for testing later

Initial Steps

Not knowing what 360Maas is, I entered my details like a guinea pig, sharing all my personal information across the interweb. I trust the site is checked out, and legitimate.

I have the following questions early on:

1. what is the service here?

2. how does it work?

3. can i get something working?

Lets go ahead with 1.

MaaS360 - Just realised I am being thrown in at the deep end here with no context. I have no idea what this company does, but I am asked to give first impressions. This is unfair, as my level of interest in the company, its website and its products is currently zero. I will do 5 mins of research to find out what the company does and what I stand to benefit from it. Then I will know my motivations and my observations can be balanced from that. What this means is, if I stand to gain from using this tool, then my observations are probably more valuable than if I dont stand to gain, as my interest will be lower in the 2nd case.


As with any testing, the configuration of your set up is important. Any one of those variables could be a hidden variable and affect bug observation. therefore, my setup is:

OS: Win7 32b

Browser: Chrome

Last restart: about 5 days ago

Issue: country does not populate in profile

Country did not populate in my profile

I chose an arbitrary timezone, as Sydney was not available. I was able to correct this when I logged in and edited my profile.

Information sources

registration email

Issue: Security from email to website login

The website appeared to not ask me to login from the link in the email. Instead it assumed my login. I was auto logged in, and allowed to continue. This is a possible security breach. For a company that must put security first, this is a major inconsistency with that image they are trying to portray.

Issue: Feedback Popup page

When I enter in a long message by copying and pasting, the page freezes. Im not sure on the platform it is operating on but now I am interested. What other such platform restrictions are there? Is the website too busy to handle large requests? I would imagine the company would like to have a product that is efficient, as it needs to handle a lot of traffic and configuration changes and application uploads and downloads. If it freezes, then the usability and operability goes out the window, and the customer will leave.

Learning checkpoint

Ok, now I know what the site  does. I get it. It manages mobiles devices like Blackberry enterprise server.

Outstanding questions:

1. How do i register a device to see it working?

2. Is this in scope of this exercise?

3. Am I willing to register my device? considering the usability issues I have seen so far.

Issue: My support center downloads are automatic

I am interested in downloading some more products or seeing what is on offer.

when I click the link Products, an automatic download begins. I am not confident I want this yet, but it is forced upon me. I stopped this immediately.

Tour/Demo: download an android google play app to my phone from the portal

Issue: they call google play, android market. That is the old name. Needs updating.

Getting started steps

Issue: The first step seems to be requiring me to download my apple MDM certificate. I dont have one, and dont have an apple device. Do I need one for this service? Did I miss something. I just want to link my android device and download an app.

Can I not continue now?

I have moved on to the next step in the process: Step 2

Issue: its odd that I have to enter my password now. Does the portal not account for the fact that I am logged in? Are there different password levels? Does one user have multiple passwords? I dont get this.

I am out of time. I would like to have completed my first tour and extracted the usability issues from it for consideration. As I learned from my first tour, I would come up with more tours. Once I did enough tours, I would then launch into preparing a set of test ideas.

I would also be in a position to do more paths and check more features. It is important to work out the oracles I would use. they are not clear.

From here, I would then seek feedback on the relevance of each of them, the risks associated, and the rationale for their importance. I can structure my tests better knowing this information.

But, alas I am out of time.