Significant digits

[geoff-h -at- MTL -dot- FERIC -dot- CA]

 Jennifer Kraus wondered about the use of "significant
 digits" and gave the following example of her boss'
 suggestions:

 ***********
 250 gallons = 940 L and not 946 L
 500 gallons = 1890 L not 1893 L
 30-125 psi = 2.1-8.62 bar not 2.06-8.62 bar
 ***********

 The basic notion is useful, but your boss's implementation
 is wrong. You can see this quite easily by comparing your
 first and second examples (gallons): your boss's version is
 off by 10 L, versus only 1 L for your version (based on the
 correct conversion factor). This tenfold difference is
 likely significant in some or perhaps even many cases, even
 though the magnitude of the difference (ca. 10/1000 = 1%)
 may look small. The rule for conversions is to use as many
 figures as you have available in the conversion factor, and
 round only the end result, never the conversion factor. The
 last example is also wrong because the boss is using
 different numbers of significant figures for the two ends
 of the range, a definite no-no. Worse yet, the difference
 between the boss' value and the actual value is now closing
 on 2%, which is beginning to get serious.

 <<A lot of our literature expresses flow rates in gallons
 per minute...commonly without using a decimal. So do I just
 round up the liters?>>

 Depends. The most important point in any conversion
 (ignoring the pure mathematics for a moment) is the
 magnitude of the smallest difference that is significant
 for your users. If they won't notice differences of +/- 1
 L, then you can safely round to the nearest liter. If, on
 the other hand, a 0.01 L difference is important, than you
 have to provide values accurate to the nearest 0.01 L. You
 also have to consider the cumulative effect of the
 difference. Think of a basement pump in flood-prone areas:
 if the pump puts out 0.3 gpm less than the rate of incoming
 water, that's 18 gallons per hour... so you certainly
 couldn't round to "0 gpm", even though strict mathematics
 would permit this. This is the sort of information you can
 put in your style guide, but you've got to understand your
 audience's needs.

 <<I have two separate flow rates listed in two different
 areas of a performance data sheet. One flow rate is 0.6
 gpm, the other is 1 gpm.>>

 This is _highly_ questionable if both describe the same
 process. The difference between the two values would then
 be 66% (1.0-0.6, all divided by 0.6), and unless your
 equipment is very low precision, a difference this large is
 unacceptable. But again, if nobody will notice a difference
 of +/- 1 gpm, then you could probably round to the nearest
 whole gallon. If not, don't round.

 <<I may be making a mountain of a molehill here...>>

 Quite on the contrary. Your boss is making a mountain into
 a molehill, with potentially disastrous consequences for
 anyone who tries to fly a plane over the ostensible
 "molehill". If you need some ammunition for correct use of
 significant digits, check any good statistics textbook
 under "significant figures". But remember, there are always
 two kinds of significant figure: the mathematical ones, and
 the ones that are significant to your audience. Opt for the
 latter if you have a choice.

 --Geoff Hart     @8^{)}     geoff-h -at- mtl -dot- feric -dot- ca
 Disclaimer: Speaking for myself, not FERIC.

 TECHWR-L (Technical Communication) List Information: To send a message
to 2500+ readers, e-mail to TECHWR-L -at- LISTSERV -dot- OKSTATE -dot- EDU -dot-  Send commands
   to LISTSERV -at- LISTSERV -dot- OKSTATE -dot- EDU (e.g. HELP or SIGNOFF TECHWR-L).
  Search the archives at http://www.documentation.com/  or search and
browse the archives at http://listserv.okstate.edu/archives/techwr-l.html