Asking good questions of your users is essential to know how your customers use your documentation. This is part 2 after my post about using a survey to get to know your audience. The introductory paragraphs are identical to last week’s post, so feel free to skip ahead to the two paragraphs before the next heading.
A recent thread at Technical Writing World got me thinking about user surveys and revisiting two posts from 2010 where I wrote about obvious, but not so helpful questions and how to segment users and survey your documentation.
You can get all kinds of answers from your users, but depending on your questions, I find some answers more helpful than others. Jakob Nielsen’s position about usability makes a lot of sense to me:
Pay attention to what users do, not what they say. Self-reported claims are unreliable, as are user speculations about future behavior.
But if we cannot observe users, we have to rely on what they say – and try to ask them questions which yield useful answers. If you include an odd number of set answers in your survey, you can slice up the results (instead of having to deal with freewheeling answers), and users have a chance to select the option in the middle if they want to.
– Here are some questions that can help you to find out how successful your documentation is. Note that it doesn’t simply ask customers what they want. Instead, the questions can help you to determine which deliverables and which topic types (such as concepts and procedures) need improvement.
There are obviously other and better ways to determine user behavior, from usage analytics to user-generated content – but many technical writers don’t have the mandate or means to simply use them. These ideas are for them.
How do you use the documentation?
Do you use the documentation…
- To set up & configure <our product>
- To operate <our product>
- To learn about the business
- Not at all
When you need help, where do you normally seek information?
- I look in <product> online help
- I look in <product> user manuals
- I ask a colleague
- I call customer services
- Other
How frequently do you use <our product> online help?
- Daily
- A few times a week
- A few times a month
- Once a month or less
- Never
How frequently do you use <our product> user manuals?
- Daily
- A few times a week
- A few times a month
- Once a month or less
- Never
How successful are you with the documentation?
Can you find the information you need in the documentation quickly and easily?
- No, not at all.
- More or less.
- Yes, very much so.
Is the documentation that you find accurate and complete?
- No, not at all. | More or less. | Yes, very much so.
Is the documentation clear and easy to understand?
- No, not at all. | More or less. | Yes, very much so.
Does the documentation help you achieve your task?
- No, not at all. | More or less. | Yes, very much so.
Does the documentation provide sufficient theoretical and business background?
- No, not at all. | More or less. | Yes, very much so.
Are you satisfied overall with <our product>’s documentation?
- No, not at all. | More or less. | Yes, very much so.
Your turn
Which questions do you use to find out whether your documentation hits the spot? Feel free to leave a comment.
Kai's Tech Writing Blog 
Great post Kai and a comprehensive list of questions to assess the usefulness or otherwise of documentation from the customer’s perspective. I typically include some open questions at the end, for example, “please let us know about any specific parts of the doumentation that we could improve” or “how could we enhance our documentation to make life easier for you”. While not everybody takes the time to answer open questions, those who do can provide invaluable insights.
Thanks, Patrice. Indeed, the surveys I’ve participated in also included open questions. They surely don’t hurt and yield good results, from the occasional typo to use cases we’d never thought of.
“There are obviously other and better ways to determine user behavior, from usage analytics to user-generated content – but many technical writers don’t have the mandate or means to simply use them.”
It seems just as likely if a technical writer has the ‘mandate’ to create and analyze a survey that they could collect analyze metrics such as clicks or views for documentation. In all, I wonder which is the more useful information though (if you can only do one or the other).
Thanks, Fer, for your comment.
Actually, in many (European) settings I’ve seen, it can be a lot easier to do a survey than to collect web site analytics, for technical and legal reasons. For documentation in CHM, PDF or print, there’s often awfully little to analyze.
I think the information and benefits from them is complementary. Some tech writers I know are just glad to get to do one of both, so they don’t exercise a choice, but simply take what they can get.
[…] Find out how users use your documentation […]
Great post Kai.
A good questionnaire can certainly ensure you know how your documentation is being used. The responses can also help ensure the team/author validate the usefulness of the documents you produce and the information within each artefact.
Let me tell you my questionnaire story…
A few years ago the organisation I work for moved toward a more Agile development model, rolling it out on a major project. I joined the project a couple of months after its inception and being new to Agile was impressed when told there had been a number of documentation discussions and a documentation spike had been generated. I was less enthused when said spike turned out to be a rather sorry looking post-it note with my initials and the word Documentation written on it. The entire project team seemed to have been brainwashed and were all chanting a’documentation light’ mantra. At that point it seemed to mean that no permanent supporting documentation was planned for the new functionality and I was to justify the creation and maintenance of each required document.
I decided to go direct to the information stakeholders and ask them what they needed from the project in order to perform their work as and when they joined the project or encountered the delivered functionality. I put together a reference group from many teams across the organisation: Support, Maintenance, Development, Designers, Consultants, Trainers and QA colleagues. My questionnaire drew from them the types of information they need to perform their task and where they normally go to get such information (such as a particular document or SME).
We cross-referenced the results against the documentation artefacts we were producing at that time and found that we could meet the information requirements of our users with a smaller set of more focussed documents, reduce duplication and remove redundant information.
The understanding we obtained from that original questionnaire helped us build a set of documentation artefacts that have changed little in over four years. I still use the cross-reference chart to ensure we put the correct information into each document and are still meeting those original requirements.
And as for analytics…I’d love to be able to examine what Help topics my readership are visiting but cannot do that the way ourHelp is deployed. We do have an email feedback link on each topic and from that we have received some useful feedback. We have even received a couple of purely complementary mails – I nearly had to go for a lie down!