meeting goals/benchmarking

[Steven Jong <SteveFJong -at- AOL -dot- COM>]

Hope Creskoff <hac -at- LURCH -dot- CORP -dot- HARRIS -dot- COM> asked about metrics for
documentation groups. I would be concerned, first, that her two-person group
is too small for measurements, because they would tend to point to
individuals, and I think individuals should never be measured.

Her comment that measuring developer productivity by lines of code written
"has been proven to be an ineffective way to measure a developer's
productivity" surprised me. Have Tom DeMarco and Watts Humphrey suddenly
stopped advocating that metric? I don't think so. That metric is alive and
well, though admittedly the engineers are kicking.

Hope is "not very comfortable" with measuring page output, because:

"1) we may need to use more graphics/notes/etc. to document one document or
another and 2) we are now producing online docs in HTML - 'pages' don't
really apply."

I would ask how an HTML page differs from a printed page for the purpose of
collecting the metric. I'm not at all sure that's a valid objection. But
that's a tactical response, not a strategic one.

Having studied the question of documentation metrics for several years now, I
am always struck by how our reaction is identical to the reaction of
professionals in all other fields when the question of quality improvement is
put to them. "That's all very well," they say, "but *our* work is different,
and cannot be measured." Seriously--everybody says that! After a while, one
begins to take that reaction with a grain of salt, and look for ways in
which, just perhaps, the work might *not* be different, might *not* be
unmesaurable.

I think the concerns over measuring writer productivity by pages produced
stem from a confusion of *product* quality and *process* quality. The two are
different--two different dimensions, if you will--and you can't mix metrics
between them. You must distinguish between product quality--that is, are you
producing good stuff?--and process quality--that is, are you producing stuff
good (well)? To me, product quality is always the more important thing. To
measure product quality, customer satisfaction surveys (which Hope's company
doesn't want to use) certainly *are* acceptable, if subjective, metrics. (The
key result expected could, for example, be an average 4 out of 5 on a
satisfaction survey.)

I would argue, without going into specifics, that we can all agree there is
such a thing as document quality, and that we know it when we see it. Call it
Q. Certainly Q is not a function of the number of pages; I can't directly
make the book better by making it longer or shorter. However, given two
documents of equal quality, one can then measure them, along another
dimension, by process quality. (Call it P.) Is is reasonable to note that one
document was created by one writer, on time, under budget, and with a minimum
of nights, weekends, and general angst, while the other document took a
writer, an emergency contractor, three editing passes, four weekends and an
80-hour final week, and a rush-job printing. Which book is better? I'd say
the one that was more smoothly created, even if I can't tell by looking at
the final results.

All things being equal, then, it *is* reasonable to compare two writers
(though I prefer to speak of two processes) whose output is equal in quality
(Q), but whose process (P) is different. In that case, the writer/process
that produces 300 high-quality pages a year is more productive than the
writer/process that produces 200 high-quality pages a year. Hey, it's a
fundamental question management has the right to ask; and they ask it of
writers or software engineers as well. (Why else would you ever deserve a
raise?)

What, then, are good process metrics? Again, a metric Hope's company
apparently doesn't care about would by my #1 choice: adherence to milestones.
Completing a project on time and under budget is a fundamental measurement,
whether the product is a user's guide, a software application, a bridge, or a
nuclear submarine. (Budget for writing projects is almost entirely writer
time, so time *is* money for us.)

I would not advocate any one single metric to assess documentation product or
process quality; that's insufficient and lazy (to address Robert Plamondon's
subsequent, pungent comment 8^) However, by taking a series of measurements,
one can arrive at an accurate picture of what you're doing and how well
you're doing it.

To comment on Alexia Prendergast's subsequent comment about counting
editorial changes per page, is that really a useless metric? Is there really
no difference between a writer whose work requires heavy grammatical and
style editing and one whose drafts are right the first time (whatever "right"
means in a given shop)? Sure there is, and I know it even if the final
versions are identical. A magazine I saw on software testing had an article
on metrics, which advocated the metric "rate of change of bugs filed against
product." I thought that was cool--and it's based on the same idea, once you
make the translation. Why is that metric useful for measuring code quality
but useless for measuring documentation quality?

In conclusion, I think you must distinguish between product quality and
process quality. Once you do that, if you know what's important to you, you
can measure it.

          -- Steve

=============================================================
Steven Jong, Documentation Group Leader ("Typo?  What tpyo?")
Lightbridge, Inc, 281 Winter St., Waltham, MA 02154 USA
<jong -at- lightbridge -dot- com>, 617.672.4902 [voice], 617.890.2681 [FAX]
Home Sweet Homepage: http://members.aol.com/SteveFJong

 TECHWR-L (Technical Communication) List Information: To send a message
to 2500+ readers, e-mail to TECHWR-L -at- LISTSERV -dot- OKSTATE -dot- EDU -dot-  Send commands
   to LISTSERV -at- LISTSERV -dot- OKSTATE -dot- EDU (e.g. HELP or SIGNOFF TECHWR-L).
  Search the archives at http://www.documentation.com/  or search and
browse the archives at http://listserv.okstate.edu/archives/techwr-l.html