Improve documentation with quality metrics

Quality metrics for technical communication are difficult, but necessary and effective.

They are difficult because you need to define quality standards and then measure compliance with them. They are necessary because they reflect the value add to customers (which quantitative metrics usually don’t). And they are effective because they are the only way to improve your documentation in a structured way in the long run.

Define quality standards

First, define what high quality documentation means to you. A good start is the book Developing Quality Technical Information: A Handbook for Writers and Editors from which I take these generic quality characteristics for documentation topics:

  • Is the topic task-oriented?
    Does it primarily reflect the user’s work environment and processes, and not primarily the product or its interface?
  • Is the topic up-to-date?
    Does it reflect the current version of the product or an older version?
  • Is the topic clear and consistent?
    Does it comply with your documentation style guide? If you don’t have one, consider starting from Microsoft’s Manual of Style for Technical Publications.
  • Is the topic accurate and sufficient?*
    Does it correctly and sufficiently describe a concept or instruct the customer to execute a task or describe reference information?
  • Is the topic well organised and well structured?*
    Does it follow an information model, if you have one, and does it link to relevant related topics?

* Measuring the last two characteristics requires at least basic understanding of topic-based authoring.

The seal of quality

You may have additional quality characteristics or different ones, depending on your industry, your customers’ expectations, etc. As you draft your definition, remember that someone will have to monitor all those characteristics for every single topic or chapter!

So I suggest you keep your quality characteristics specific enough to be measured, but still general enough so they apply to virtually every piece of your documentation. Five is probably the maximum number you can reasonably monitor.

Measure quality

The best time to measure quality is during the review process. So include your quality characteristics with your guidelines for reviewers.

If you’re lucky enough to have several reviewers for your contents, it’s usually sufficient to ask one of them to gauge quality. Choose the one who’s closest to your customers. For example, if you have a customer service rep and a developer review your topics, go with the former who’s more familiar with users’ tasks and needs.

To actually measure the quality of an online help topic or a chapter or section in a manual, ask the reviewer to use a simple 3-point scale for each of your quality characteristics:

  • 0 = Quality characteristic or topic is missing.
  • 1 = Quality characteristic is sort of there, but can obviously be improved.
  • 2 = Quality characteristic is fairly well developed.

Now, such metrics sound awfully loose: Quality “is sort of there” or “fairly well developed”…? I suggest this for sheer pragmatic purposes: Unless you have a small number of very disciplined writers and reviewers, quality metrics are not exact science.

The benefit of metrics is relative, not absolute. They help you to gauge the big picture and improvement over time. The point of such a loose 3-point scale is to keep it efficient and to avoid arguments and getting hung up on pseudo-exactitude.

Act on quality metrics

With your quality scores, you can determine

  • A score per help topic or user manual chapter
  • An average score per release or user manual
  • Progress per release or manual over time

Areas where scores lag behind or don’t improve over time give you a pretty clear idea about where you need to focus: You may simply need to revise a chapter. Or you may need to boost writer skills or add resources.

Remember that measuring quality during review leaves blind spots in areas where you neither write nor review. So consider doing a complete content inventory or quality assessment!

Learn more

There are several helpful resources out there:

  • The mother lode of documentation quality and metrics is the book Developing Quality Technical Information by Gretchen Hargis et al. with helpful appendixes, such as
    • Quality checklist
    • Who checks which quality characteristics?
    • Quality characteristics and elements
  • Five similar metrics, plus a cute duck, appear in Sarah O’Keefe’s blog post “Calculating document quality (QUACK)
  • Questionable vs. value-adding metrics are discussed in Donald LeVie’s article “Documentation Metrics: What Do You Really Want to Measure” which appeared in STC’s intercom magazine in December 2000.
  • A summary and checklist from Hargis’ book is Lori Fisher’s “Nine Quality Characteristics and a Process to Check for Them”**.
  • The quality metrics process is covered more thoroughly in “Quality Basics: What You Need to Know to Get Started”** by Jennifer Atkinson, et al.

** The last two articles are part of the STC Proceedings 2001 and used to be easily available via the EServer TC Library until the STC’s recent web site relaunch effectively eliminated access to years’ worth of resources. Watch this page to see if the STC decides to make them available again.

Your turn

What is your experience with quality metrics? Are they worth the extra effort over pure quantitative metrics (such as topics or pages produced per day)? Are they worth doing, even though they ignore actual customer feedback and demands as customer service reps can register? Please leave a comment.


Index this!

An index is an important navigation device in documentation, especially in print. It helps users to quickly find key terms, concepts, functions, and instructions.

In this post, I share my best advice about building an index. And I ask for your help and opinion on a couple of details.

To make your index entries helpful, anticipate user searches. Think of it as a parallel to online search results. When building an index, think primarily of what users might want to look up and find in your documentation.

To create an index, follow these three steps:

  1. Create index entries.
  2. Build the index.
  3. Clean up your index.

1. Create index entries

  1. Scroll through your entire document and search for words and phrases that you want to appear in the index. Select only occurrences with substantial descriptions, not just mere mentions.
  2. Create an index entry, depending on your help-authoring tool or other software. Edit the entry to avoid all unnecessary words, such as articles (“a” or “the”) and prepositions (“in” or “by”, etc.).

When creating index entries, consider these issues:

  • Redundant entries, where you have several entries for the same thing, may seem wasteful and confusing. However, if users are likely to search for the word “equity” in addition to “stocks”, consider using one of the terms to refer the reader to the other entry.
  • Cross-references between index entries is a suitable compromise between usability and completeness: Of course, it is an extra step for the reader to move from the entry “Equity: see Stocks” to the “Stocks” entry. On the other hand, it is much more difficult for writers to keep both entries “Equity” and “Stocks” complete and identical over time. Which do you prefer, as a reader and as a writer: Is it okay for readers to move to a second entry? Or should the page numbers be at each entry?
  • Nested entries can help you to structure your index and alert the user to hierarchies of concepts and functions. It also cleans up your index by avoiding redundant words. Don’t have more than two levels, however. What’s your experience: Are nested entries helpful or generally confusing to readers?

2. Build the index

At any time after creating your first index entries, create and preview your index, depending on your help-authoring tool or other software.

3. Clean up the index

After you have created all index entries, rebuild and update your index. Then review the existing index for inconsistencies and go back to the index marks in the text to correct the following issues:

  • Omit double entries of the same term, but with different spelling or singular and plural.
  • Break up index entries that list more than 5 target pages. Refine index entries, for example by creating additional second-level entries for separate tasks, such as “Setting up”, “Calculating”, etc.

Optimal length and scope of an index is the result of a compromise: Make your index as long as necessary to support readers to use your documentation quickly and efficiently. And make your index as short as possible to spare yourself unnecessary work and the reader confusion.

Whether your index is too long or too short is a matter of judgement. Put yourself into the reader’s shoes once more and review your index. You may find that your index is a little too short and missing some terms that users might also look up.

Read more

Good resources for building an index are these two web pages and three books:

Your turn

Please help me out by weighing in with your experiences and opinions to the questions above. Or feel free to tell me if  I forgot anything? How do you create an index for printed documentation? Should online help also have one, or is the search enough? Please leave a comment.