Comparing Engagement Survey Results Year Over Year: Which Changes Are Really Meaningful?

Comparing engagement survey results year over year is the most direct way to see whether the steps you have taken are working. But not every difference is real change. For a difference to be meaningful, two conditions must hold: the two measurements must be genuinely comparable (the same items, the same scale, the same calculation method and a similar employee population), and the difference must be too large to be explained by chance once group size is taken into account. In this article we explain how to check both conditions, the factors that distort the trend and how to report the results.

Prerequisites for Comparison

Before putting two years' results side by side, make sure the measurements measure the same thing in the same way:

If one of these conditions has changed, comparison may still be possible, but the change should be noted in the report and differences interpreted more cautiously.

Keep a Core Set of Items

Surveys evolve over time: new topics are added and items that do not work are removed. This is natural, but changing items every year makes comparison impossible. The solution is to keep a core set of items that stays unchanged:

• The items that make up the engagement index should never be changed.
• At least two or three items in each category should stay the same for several years.
• New topics (such as hybrid work or AI use) can be added as extra modules without touching the core set.
• If you must change an item's wording, break that item's trend line and start a new baseline with the new wording. Do not compare it with the old result on the grounds that "it was a similar item."

Designing items well from the start reduces the need to change them later. Our article on common survey question mistakes can help. If you are considering changing survey providers, be sure to ask whether the new provider can keep your core items.

Which Differences Are Meaningful?

Group Size Is Decisive

In every measurement, results fluctuate somewhat by chance. The smaller the group, the larger that fluctuation. The table below shows roughly how large a difference between two years must be, for an item with a favorable response rate around 50%, to be distinguished from chance with 95% confidence:

The values are a rough guide calculated with the classic method for a single item with a favorable response rate around 50%. The thresholds get somewhat smaller as the rate moves away from 50%. The engagement index, which averages several items, also fluctuates less than a single item.

In practical terms: an 8-point drop since last year in a unit of 100 may look striking, but it may be down to chance. By contrast, a 3-point drop in the engagement index of an organization of 5,000 most likely signals real change.

Testing the Difference

For a more precise assessment of whether a difference is meaningful, a significance test for the difference between two proportions is used:

z = (p₂ − p₁) / √[ p₁(1 − p₁) / n₁ + p₂(1 − p₂) / n₂ ]

Here p₁ and p₂ are the favorable response rates for the two years, and n₁ and n₂ are the numbers of respondents. If z is greater than 1.96, the difference is considered meaningful with 95% confidence.

Example: Suppose an organization's engagement index was 64% last year (820 responses) and 68% this year (850 responses).

• The square root in the denominator: √(0.64 × 0.36 / 820 + 0.68 × 0.32 / 850) ≈ 0.023
• z ≈ 0.04 / 0.023 ≈ 1.73

Because z is below 1.96, the 4-point increase is not meaningful with 95% confidence. This does not mean there was no increase; it only shows that in an organization of this size a 4-point difference could also be explained by chance. If the increase continues in the next measurement, the trend becomes clear.

In surveys with very high participation, some analysts apply a correction (the finite population correction) because almost all employees responded, which lowers the thresholds. However, since employees' views fluctuate over time even when everyone responds, we recommend the uncorrected, more cautious approach for year-over-year comparisons.

Statistical Significance vs. Practical Importance

Not every statistically significant difference is important. In an organization of 20,000 people, a 1.5-point difference may be statistically significant yet mean little in practice. Roughly:

• 1–2 points company-wide: Usually read as "no change."
• 3–5 points: A change worth attention if it is also statistically supported.
• 5 points or more: A clear change whose causes should definitely be investigated.

Three Effects That Distort the Company-Wide Average

1. The Composition Effect

The company-wide average can change even if no group's result gets worse. For example:

The values in the table are for illustration only.

Although engagement rose by 2 points in both groups, the company-wide average fell by 2.5 points. The reason is that the share of frontline employees, whose engagement is lower, has grown within the organization. Leadership that misses this may read an improving picture as deterioration. Before interpreting a company-wide change, always look at the change in the main groups as well. We explain how to build groups in our article on segmentation in employee surveys.

2. The Participation Effect

When the response rate changes markedly, the population that responds changes too. For example, if participation rose from 60% to 85%, employees who did not respond last year have taken part this year. Because employees who do not take part in surveys are usually less engaged, higher participation can pull results down without any real deterioration. Conversely, if participation fell, results may look better than they are because they reflect only the responses of more engaged employees.

To separate this effect, look at how participation changed by group: is the drop in results concentrated in the groups where participation rose most? For more on participation, see our article on employee survey response rates.

3. External Events and Timing

Events during the survey period directly affect results: layoffs, mergers, a change in senior leadership, the announcement of pay increases or economic uncertainty. Keeping the survey period similar every year and noting major events of that period in the results report helps interpret differences correctly.

Comparing Units When the Organization Changes

Restructurings are the biggest obstacle to comparison at unit level. The recommended approach:

1. Create a mapping table: Identify which unit in the new structure each unit in the old structure corresponds to.
2. Compare only comparable units: Show a trend for units where most employees have stayed the same.
3. Accept a new baseline: For units that merged, split or saw most of their employees change, do not compare with last year; this year's result is the first measurement of the new structure.
4. Note manager changes: Even if the unit stayed the same, if its manager changed, the change may largely reflect the new manager's influence.

In anonymous surveys employees are not tracked individually; comparisons are made between groups, not individuals. So a change in a unit may partly come from employees joining and leaving that unit. We explain the differences between anonymity models in our article on anonymity in employee surveys and KVKK (Turkey's Personal Data Protection Law).

What to Look At: The Right Order of Comparison

1. Check comparability: Are the items, scale, coverage and timing the same?
2. Compare participation: How did participation change overall and by group?
3. Look at the change in the engagement index: Is the difference meaningful given the group size?
4. Drill down to the main groups: Does the company-wide change come from changes within groups or from a change in the groups' weights?
5. Look at the areas where action was taken: Has anything changed in the areas prioritized last year? This is the most direct indicator of whether the steps taken worked.
6. Identify the items that changed most: List only the meaningful differences.
7. Look at the distribution: Even if the favorable rate stays the same, a rise in unfavorable responses may signal growing polarization.
8. Validate with open-ended comments: If a topic comes up more often in comments than last year, the change becomes clearer.

We explain how to identify priority areas in our article on key driver analysis in engagement surveys.

Not a Single Year, but a Trend

The difference between two measurements is a single data point. Three or more measurements make it much easier to tell a real trend from chance: a result that rises by 2 points three years in a row is a more reliable improvement than one that jumps 6 points in one year and falls back the next.

Short pulse surveys between annual surveys allow you to track the trend more often. But pulse surveys also need to use the same items, and small differences should not be overstated. We explain the measurement calendar in our article on how often to run employee surveys.

Reporting the Results

• Show the number of responses next to every result: Readers can then judge for themselves how reliable a difference is.
• Flag only meaningful differences: Put up or down arrows only on differences that pass the threshold; show the rest as "no change."
• Mark breaks in the trend: For results whose item, coverage or unit has changed, break the trend line and note why.
• Add context: Include major events in the organization and changes in participation and composition in the report.
• Do not rank managers by change: In small teams, change may largely be down to chance; ranking leads to unfair conclusions and to pressure in the next survey.

Year-over-year comparison becomes stronger when read together with external comparison: if the organization's results are falling while the general trend in the sector is also falling, the picture should be interpreted differently. See our article on employee engagement benchmarks. If your results have clearly worsened, you can use the roadmap in our article on what to do when employee survey results are bad.

Common Mistakes

• Comparing changed items: The result of an item whose wording or scale has changed cannot be compared with the previous year.
• Overstating differences in small groups: A 10-point change in a team of 30 is often down to chance.
• Overlooking composition and participation effects: A change in the company-wide average is not always a change in employees' experience.
• Looking only at the company-wide average: The average may stay flat while one group improves markedly and another worsens.
• Setting new priorities every year: Moving to new priorities without tracking the change in the area where action was taken for at least two or three measurements prevents you from learning what works.

Frequently asked questions