Usability Testing -- Our Experience

[Trevor Grayling <TREVOR -at- MDLI -dot- COM>]

James:

With reference to your message:

-----------------------------------

Date:    Mon, 17 Oct 1994 17:36:39 EDT
From:    James Brown <jimb -at- SSINY -dot- COM>
Subject: Usability Goals

I'm designing the documentation set for a new product and must
establish some usability goals for the documentation.
These goals are supposed to help establish guidelines for future testing.
Does anyone have experience with setting usability goals and then testing?
What are the pitfalls? Send me your war stories.
Jim Browne
------------------------------------


We defined goals for a new documentation set some time ago. We usability-
tested against the goals, made some minor changes, and then tested again.
The results were positive, and this was confirmed by a later statistically-
significant survey of end users.

The documents we tested were a Tutorial and a task-oriented Reference for a
Windows-based drawing program for drawing chemical structures.  The goal of
the documents, as defined in the doc plan, was to get the average user "up
and running" and able to draw the chemical structures they were interested
in with only occasional reference to the Reference manual for those more
obscure functions not covered in the Tutorial.

Given the goal, we devised the following usability test:


1. REPRESENTATIVE AUDIENCE: We found seven users in-house who represented
the skill sets of our end users: Some had no experience at all with previous
versions of our products, some had limited experience, and some were experts
with the earlier versions.  They also had varied chemistry backgrounds.

2. DEFINING TEST CRITERIA: Since the goal was to be able to "draw structures
with only occasional reference to the Reference manual. ...", we created
four "average" chemical structures, three of which did need reference to the
Reference manual for some specific functions.  A successful test would be
one where ALL the users would be able to EXACTLY duplicate the structures
given them on paper.  It is important that the test criteria are defined
before the test (!) and are very specific. There was no time limit because
time was not part of the goal of the documentation set. How they used the
documentation was also not part of the test criteria.

3. CONDUCTING THE TEST: We set up seven computers in a conference room. We
provided each user with the Tutorial, the Reference, and the four chemical
structures on paper.  We told them: (a) that this was a test of the
documentation and not of the testers to reduce anxiety; (b) that they were
to exactly duplicate the paper structures on screen in terms of exact bond
length, bond angles, atom symbols, and so on; (c) that they could use any of
the documentation they saw fit but that they were not obliged to use it if
they didn't want to; (d) there was no time limit; (e) that we would be
looking over their shoulders and taking notes; (f) that only if they were
well and truly stuck and unable to proceed could they ask for assistance.

TEST RESULTS: All users did complete all four structures.  Time for the test
varied from 40 minutes to 1-1/2 hours.  Some dutifully worked through the
Tutorial first, some only referred to it when they got stuck, and some
skimmed through it looking for the "gist" of the product. Two users got
temporarily stuck but we refused to help them, telling them to rely on the
materials provided (we could see that they weren't really stuck). We
obtained tons of useful information by observing the users, even though the
test was successful.  Although we had ourselves tested the documentation
thoroughly, we noted paragraphs of text and instructions that were correct
but ambiguous: If something could be read in two different ways, it would
be. We also noted the need to put the "getting out of trouble" section much
nearer the front of the Tutorial.

We made the corrections noted and tested again on six different subjects
with equal success.

Hope this helps,

Trevor Grayling
Manager, ISIS End-User Documentation
MDL Information Systems, Inc.
trevor -at- mdli -dot- com