Practical usability testing?

["Hart, Geoff" <Geoff-H -at- MTL -dot- FERIC -dot- CA>]

Amy Janczy has been asked <<...to perform usability testing for the
documentation for our main product, which is a plug-in for AutoCAD, used
primarily by GIS professionals. I hope to have onsite access to at least a
few users in the immediate area... Unfortunately, I've been here less than 2
months and haven't yet had the opportunity to meet any users, or even sit in
on training sessions, although I hope to do that in the next few weeks...
I'm at at loss for how to start.>>

Usability is too large and complex a subject to cover adequately in a brief
e-mail message; if you have a university library anywhere nearby, head over
and spend a day reading up on the literature. Have a look at anything done
by J.C. Redish and J.S. Dumas, and try to get a copy of the following
article through interlibrary loan: Craig, J.S., 1991. Approaches to
usability testing and design strategies: an annotated bibliography.
Technical Communication (2nd Quarter 1991): 190-194. If you don't have good
library resources available, ask around (including on this list) until you
find nearby STC members who can provide access to back issues of Technical
Communication or the annual conference proceedings; the proceedings are
particularly good because they provide a range of practical articles all in
one convenient package.

There are a variety of approaches to testing, ranging from unstructured
("OK, dude... have a look at the documentation, and see if you can figure
out how to draw me a map") to highly structured ("turn to page 15 of the
tutorial and use the instructions there to draw the map described on those
pages"). Which one works best depends on what your goals for the usability
testing are, so you'll have to start by defining those goals. Talk to
trainers and support staff at your company and find out what work has
already been done, and whether there are any guidelines for how the company
does this kind of work. They may also help you set goals based on the
problems they've already encountered (e.g., "we get far too many calls
asking us to explain how to install the product... could you figure out
what's wrong with the installation documentation?"). Sample goals: minimize
the time spent finding and using instructions, increase the accuracy of
following those instructions, minimize the need for instructions in the
first place, change your vocabulary to better match the reader's knowledge,
etc. List your goals, and prioritize them (with the help of your
colleagues).

While you're doing this preliminary research and waiting for the reading
material to arrive, contact the users of your product and begin building
bridges with them. That's particularly important, because when you arrive to
do the tests, you don't want to be a scientist watching lab rats who know
that they're only expendable lab rats: you want to be a human being who's
established a connection with another human being. You want to make it clear
that you're not testing _them_, but rather that _they're helping you_ to
produce better documentation; given how stressful it can be trying to use
unfamiliar software while someone's watching you and waiting for you to
fail, you'll need to spend a fair bit of time making them comfortable. Make
sure that you're basically up to speed with the documentation and the
product before you go, because if you're not, then you're no further ahead
than the user you're trying to help, and that can cause problems.

Find out what the users hope to achieve with the product, and use these
goals as the basis of your tests. (Oddly enough, if nobody's done a proper
audience analysis thus far, these goals may not be the goals that the
developers expected. Finding this out makes you worth your weight in gold to
the company!) For example, if one of their main goals is to annotate GIS
basemaps using your plug-in, you should design a simple test that makes them
actually do this task. Plan to watch carefully how they approach the task of
using your documentation (so you can spot things that _they_ don't notice
and report), but also get them to report problems as they encounter them
(things they _do_ notice). Combine these results and you'll have a good
start to identifying general and specific usability problems. Here's an
example: you may notice that the users don't refer to the documentation step
by step, and after skimming it briefly, dive right into using the software;
that suggests they need onscreen affordances (clues to what they're doing),
and that the docs should emphasize high-level information (describing the
overall strategy) rather than minute details ("click OK"). When you're
starting out, and there's no usability data already recorded, emphasize
general aspects of the documentation with broad applicability so you can
improve the overall documentation; worry about fine points (e.g., specific
tasks covered by procedural information) only once you've got the time or if
a specific problem was identified as problematic by tech support. But don't
let your broad focus blind you to specific problems; if you do spot
something unique that you hadn't planned to investigate, then by all means
make the time to investigate that problem if you feel there'll be good
payback.

That's a very shallow and simplistic overview, with lots of holes, but as I
say, there's too much info. available to do justice to it in a short e-mail.
Write back with specific questions once you've done more reading in this
area and begun planning your on-site visits.

--Geoff Hart, FERIC, Pointe-Claire, Quebec
geoff-h -at- mtl -dot- feric -dot- ca

Hofstadter's Law: The time and effort required to complete a project are
always more than you expect, even when you take into account Hofstadter's
Law.