Should Employee Engagement Surveys Be Used in Performance Evaluations?

“Senior managers soon may have more to fret about when it’s time to discuss their performance with the boss: Are they engaging employees and creating inclusive work environments?

 

Agencies will evaluate Senior Executive Service members and other managers on those soft skills, according to a draft, obtained by Federal News Radio, of a memo to agency leaders from the Office of Management and Budget, the Office of Personnel Management and the Presidential Personnel Office.​”

via Federal News Radio​.

While it’s laudable to encourage managers to improve their subordinates’ engagement, surveys are not always an effective tool when performance evaluations, and potentially compensation, are tied to them.

Risks of small sample sizes. While the whole of government represents a very large sample, individual managerial groups have the potential to be very small. There are several dangers for relying on small samples. The first is that a single respondent can skew the aggregate responses. The chart below shows the relative impact of one respondent on a small sample size compared to a larger sample size.

Surveys of very small groups also have a lower likelihood that anonymity can be preserved. In the example above, it is unlikely that a manager of a group of 5, 10, or even 20, would not know who is selecting a 1 on a question. Qualitative questions, which are important for providing better context for question interpretation, are even more likely to risk exposing anonymity. Concerned employees who feel they may be at risk of not remaining anonymous are less likely to answer honestly in future surveys or they may skip participation altogether rather than risk potential retaliation.

Inconsistently interpreted questions. Employee engagement questions, while often predictable, but like other likert scale questions, can be open to interpretation from respondent to respondent. We know that the difference between 100 pounds and 101 pounds, is exactly one pound. And that one pound is the same from person to person. However, the perceived difference between 4 and 5 on a 5 point scale which represent feelings, are simply not the same. While it only looks like one point, that one point may mean almost nothing to one person and everything to the next person. Add a small sample size to this issue, and the scale variations could be significantly exacerbated.

Survey frequency. An annual survey, while a good start for a consistent tracking, may not be frequent enough to smooth out outliers, while monthly or quarterly could be too frequent depending on the population and create unnecessary noise in the data.

Outside factors. Government employees, like others, could find their engagement responses impacted by conditions outside their management’s control. For example, it may be a challenge for federal middle management to improve overall employee engagement when they can’t control when Congress will pass a budget or if the government will be shut down again.

From the perspective of this market researcher, surveys can be used as part of a larger evaluation toolkit, if these various factors can be accounted for and put in context. If not, then the entire exercise is at risk of failing.