Survey comments tell organizations’ ‘stories’

Analysis reveals themes, illuminates ‘whats’ and ‘whys’ of quantitative answers
By Stephanie Rohling and Bonnie Pascall
|Canadian HR Reporter|Last Updated: 11/29/2010

Employees in an organization live a shared story of culture, triumphs and trials. Management often tries to understand this story through employee surveys.

However, quantitative questions alone don’t tell the whole story, which is why it’s important to encourage and examine qualitative comments on surveys. Combined, quantitative and qualitative data provide insight into the true nature of an organization and the issues it’s facing.

Often the story that emerges from written comments will confirm for management many of the issues they knew or assumed existed within the organization, but they can also reveal new or existing issues that surprise the executive team and change how they view the organization.

The majority of employee surveys, whether they are engagement, 360-degree or issue-specific surveys, will include questions to elicit both quantitative and qualitative responses. Most often the quantitative data is the result of questions tied to a ratings scale that provide numerical or statistical data.

Effective qualitative questions should be open-ended and enrich the understanding gained by quantitative data by providing an understanding of the “what” and the “why” that cannot be easily determined through questions tied to a ratings scale.

There are two types of qualitative questions. The first is used to gain further understanding of a particular issue. The second is used to gain an understanding of an organization as a whole.

These qualitative comments only become a valuable tool to the executive team and management when analyzed effectively and used to make future change.

4 tips for getting the most out of written comments

To maximize the usefulness of written comments, the following steps should be taken:

• determine the best party or parties to analyze the comments

• cleanse the comments to protect the respondents’ confidentiality

• review comments to determine common themes that arise

• correlate the data to quantitative data.

Survey confidentiality

Having an external, third-party conduct the survey can help ease employee concerns about confidentiality. Less sensitive surveys can be executed by an in-house person who can be trusted to analyze the comments in a confidential and non-biased manner.

There are benefits to having both an external consultant and an internal employee undertake the analysis of all qualitative comments.

The external consultant’s analysis will be impartial as the consultant does not work at the organization. This impartiality allows the consultant to identify the strengths and challenges presented in the written comments and cleanse the comments to protect confidentiality prior to releasing the qualitative data to management.

Once the external consultant has analyzed the qualitative comments, the comments should be reviewed in full by at least one person within the organization who can bring context to the issues identified in the comments.

Respondents may have inadvertently identified themselves through their word choices, writing style, the substance of the comments or the use of names. While the responses should never be altered, ensuring confidentiality is key to ensuring employees’ continued confidence in the survey process. To cleanse a comment, it may be necessary to remove part of the comment that could identify the respondent or, in extreme cases, remove the comment entirely.

Qualitative comments may reveal sensitive issues of harassment, sexual harassment or assault, or other interpersonal office conflicts and misdoings that management needs to be informed of and deal with. Some comments may indicate a serious future threat to an employee or executive in the organization. This information should be removed from the data and presented to the appropriate person at the organization who can address the issue according to corporate policy and the law.

Analyzing comments

After comments have been cleansed, they should undergo a content analysis. This involves reading through the comments to determine common themes. For example, in a survey of 2,000 employees, several hundred may indicate there is no work-life balance in the organization. The common threads among the comments will start to paint a picture of the organization and its strengths and challenges. It is more important to look for themes in the qualitative comments than to focus on one or two strongly worded comments that do not fit with the themes arising from the remainder of the data.

Once the themes are identified, they should be compared to the quantitative data. Many of the themes discovered in the comments may have been explored in the quantitative section of the survey.

Quantitative data informs how strongly respondents feel about themes they have identified as important to them in their qualitative comments. This may help identify, quantitatively, how important one theme from the qualitative comments is, compared to another identified theme.

Specific questions in the quantitative data can also help management better understand an issue. For example, many respondents may indicate in their comments they are unhappy with their benefits. A focused quantitative question could show they are happy with their health benefits but unhappy because they do not receive enough flex days or vacation days.

If the quantitative and qualitative data appears to disagree, then further exploration may be required. Sometimes, a few voices show up in the qualitative comments and there appears to be a theme arising. However, the majority of respondents might have provided their opinions in the quantitative data and didn’t feel passionately enough about the issue to express their opinions in the qualitative comments.

Following the analysis of the qualitative comments, an organization should develop a plan to communicate the results to employees and use the data to make appropriate changes.

Bonnie Pascall is a senior management consultant at Pascall Management Services, an HR firm in Calgary that helps organizations attract, engage and retain talent to improve employee engagement and productivity. Stephanie Rohling is a consultant and lawyer at Pascall Management. For more information, visit www.pascallmanagement.com.

Add Comment

  • *
  • *
  • *
  • *