Written comments from employees give priceless information — but get ready for the paperwork
Most surveys contain a mixture of questions: true-false, ranking, agree-disagree. But the best ones are surveys that allow respondents to say exactly what they think.
Surveys where employees are asked to finish statements such as “In my opinion management is…” or “The best thing about working here is…,” generate the most honest, useful information. Anonymous write-in or verbatim questions as they are called are very powerful, but they create a number of problems for anyone designing or analyzing a survey.
Volume of responses
The sheer number of responses received may be overwhelming, so be prepared for the workload. Imagine dealing with a survey with three statements going out to 7,000 employees. Thousands of pages of written responses come back. The information is fantastic, but simply handling and sorting the responses is a major task.
So be careful what you ask for and know how you will handle the responses.
Analysis
If less than 100 responses are received, analysis isn’t a major problem since you can simply read them and make notes about what has been said. But if hundreds or thousands are received, a different, more cost-effective process will be in order.
One approach is to categorize the responses. Do this by reading a large number of randomly selected responses and making a list of the general categories found. The answers may pertain to management, training, communication, safety.
Now read all of the responses, keeping a record of how many times these categories appear. An “other” category should be created for those not identified the first time. This process can be extremely time-consuming, but it is the only way to quantify what people have said.
Anonymity
People will not say what they think if they fear they can be identified. Guarantee their anonymity if you want honest answers.
This can be a problem if the survey has asked them to list their location or department. In addition, some people may be concerned that others will be able to recognize their handwriting, or that their responses will be given to somebody who can identify them. One solution is to have the questions on a separate page of the survey so that they will not be recorded with location or department identifiers. As for the handwriting, either have the responses transcribed or restrict the number of people who will see the originals to a few very responsible people.
However, the easiest solution is to use an electronic, Web-based survey. This eliminates anonymity concerns.
Loss of meaning and sentiment
Categorizing or summarizing results may be necessary, but the downside is the chance that what people have said is changed or diluted. One solution is to select typical and important comments exactly as they were written.
Management deserves to hear what was said by someone who explains in detail why they love working for the organization. On the other hand, management should also see the complete statement from someone who emotionally identifies issues or concerns that are driving them mad.
Reporting results
What is the best way to report the findings? This is very simple when dealing with yes-no, ranking or agree-disagree questions that generate numbers or graphs. But it is another story when it comes to verbatim surveys.
If the answers have been categorized, use some numbers. Examples of typical responses will also enhance the report.
But what about the rest of the data? Most managers want to read everything that has been said. This can be a problem if the respondents have disclosed personal, secret or potentially damaging situation.
Surveys can include accusations about other employees that charge theft, favouritism and the giving of sexual favours. Often names are used. Deciding who should see this information requires careful thought and planning before the survey is started.
One solution is to prepare an executive summary of the findings for general distribution and reserve the complete report for senior management only.
Verbatim questions can be the most powerful, useful and provocative part of a survey. But plan ahead so that you will know how you are going to deal with and analyze the results. Be careful what you ask for. You may get more than you can handle.
John Towler is a psychologist, author and senior partner of Waterloo, Ont.-based survey and assessment firm Creative Organizational Design. He can be reached at [email protected], (519) 745-0142 or visit www.creativeorgdesign.com.
Surveys where employees are asked to finish statements such as “In my opinion management is…” or “The best thing about working here is…,” generate the most honest, useful information. Anonymous write-in or verbatim questions as they are called are very powerful, but they create a number of problems for anyone designing or analyzing a survey.
Volume of responses
The sheer number of responses received may be overwhelming, so be prepared for the workload. Imagine dealing with a survey with three statements going out to 7,000 employees. Thousands of pages of written responses come back. The information is fantastic, but simply handling and sorting the responses is a major task.
So be careful what you ask for and know how you will handle the responses.
Analysis
If less than 100 responses are received, analysis isn’t a major problem since you can simply read them and make notes about what has been said. But if hundreds or thousands are received, a different, more cost-effective process will be in order.
One approach is to categorize the responses. Do this by reading a large number of randomly selected responses and making a list of the general categories found. The answers may pertain to management, training, communication, safety.
Now read all of the responses, keeping a record of how many times these categories appear. An “other” category should be created for those not identified the first time. This process can be extremely time-consuming, but it is the only way to quantify what people have said.
Anonymity
People will not say what they think if they fear they can be identified. Guarantee their anonymity if you want honest answers.
This can be a problem if the survey has asked them to list their location or department. In addition, some people may be concerned that others will be able to recognize their handwriting, or that their responses will be given to somebody who can identify them. One solution is to have the questions on a separate page of the survey so that they will not be recorded with location or department identifiers. As for the handwriting, either have the responses transcribed or restrict the number of people who will see the originals to a few very responsible people.
However, the easiest solution is to use an electronic, Web-based survey. This eliminates anonymity concerns.
Loss of meaning and sentiment
Categorizing or summarizing results may be necessary, but the downside is the chance that what people have said is changed or diluted. One solution is to select typical and important comments exactly as they were written.
Management deserves to hear what was said by someone who explains in detail why they love working for the organization. On the other hand, management should also see the complete statement from someone who emotionally identifies issues or concerns that are driving them mad.
Reporting results
What is the best way to report the findings? This is very simple when dealing with yes-no, ranking or agree-disagree questions that generate numbers or graphs. But it is another story when it comes to verbatim surveys.
If the answers have been categorized, use some numbers. Examples of typical responses will also enhance the report.
But what about the rest of the data? Most managers want to read everything that has been said. This can be a problem if the respondents have disclosed personal, secret or potentially damaging situation.
Surveys can include accusations about other employees that charge theft, favouritism and the giving of sexual favours. Often names are used. Deciding who should see this information requires careful thought and planning before the survey is started.
One solution is to prepare an executive summary of the findings for general distribution and reserve the complete report for senior management only.
Verbatim questions can be the most powerful, useful and provocative part of a survey. But plan ahead so that you will know how you are going to deal with and analyze the results. Be careful what you ask for. You may get more than you can handle.
John Towler is a psychologist, author and senior partner of Waterloo, Ont.-based survey and assessment firm Creative Organizational Design. He can be reached at [email protected], (519) 745-0142 or visit www.creativeorgdesign.com.
