Survivorship Bias Calculator
How do you tell if an engagement score rose because unhappy people left?
Take the leavers out of last year's score, then compare only the people who were there both times. On the figures this page opens with, a score that rose from 68 to 72 rose 2.8 points because 30 people scoring 52 left, and only 1.2 because anybody's view changed. 70.6% of the rise is who answered, not how they answered. The three parts add up to the headline exactly, so nothing here is an estimate on top of an estimate.
Your engagement score went up four points. Some of the people who would have kept it down are no longer there to answer.
That is survivorship bias, and it is in almost every year-on-year engagement figure that has ever been presented to a board.
Built by Muhammad Shahzeb Anwar, MSc Industrial and Organizational Psychology, who also builds the paid tools in this shop. Most engagement reports compare two different groups of people and call the difference change. This page separates the two.
How much of the change is who answered
Any scale works: 0 to 100, percentage favourable, or a 1 to 5 average. Use the same one for every score.
Why a score can rise with nobody feeling better
An engagement score is an average of whoever answered. Between two surveys, some of those people leave, and the people most likely to leave are the least engaged ones. That is not a guess: low engagement is one of the most consistent predictors of leaving in the research, which is why organisations measure it at all.
So when the second survey comes round, the least engaged are missing from it. The average rises. Nobody who is still there has to feel any different for that to happen - the arithmetic does it on its own, by taking the low scores out of the sum.
The fix is to compare like with like: the people who answered both times, before and after. Take the leavers out of last year's figure and the comparison is between the same group of people, which is the only comparison that can show whether anybody's view actually changed.
Where to get the leavers' score
Most engagement platforms can filter last year's results by current employment status, or export them with an employee identifier you can match against leavers. If yours can, use the real figure and this page gives you an exact answer.
If the survey is fully anonymous you cannot get it, and you should not try to reconstruct it from demographics small enough to identify people. Use the range instead: put in a leavers' score 10 points below the average, then 20 below, and see how much the real change moves. If it stays positive across the range, the rise is real whatever the leavers scored.
Exit interviews are not a substitute. People leaving say what is safe to say on the way out, and it is not the same thing they would have answered in an anonymous survey a year earlier.
The same trap outside engagement surveys
Anything that averages whoever is still there has the same problem. Customer satisfaction rises after the unhappy customers churn. Average tenure rises after a round of redundancies. A course pass rate rises when the struggling students drop out before the exam. A fund's average return rises when the funds that lost money are closed and removed from the index.
In every case the figure is true, and in every case it is being read as improvement when part of it is simply who is left to measure. The question to ask of any rising average is the one this page asks: are these the same people as last time?
If they are not, the change in the figure is two things added together, and only one of them is the thing you wanted to know.
What this will not tell you
This separates who answered from how they answered, and it assumes the leavers' score you put in is right. It cannot see anybody who stayed and stopped answering: it counts them and warns you, but it cannot score them, and if they are the least engaged the real change is lower than shown. And it cannot tell you why the stayers' score moved - only how much of the change is real.
Questions people actually ask
What is survivorship bias in an engagement survey?
It is when an engagement score rises because the least engaged people have left and are no longer answering, rather than because anybody still there feels differently. The figure is true, but part of the change is who answered, not how they answered.
How do I find the average score of the people who left?
Most survey platforms can filter last year's results by current employment status. If the survey is fully anonymous, test a range instead: try a leavers' score 10 and then 20 points below the average and see whether the real change stays positive.
Does a rising engagement score mean our initiatives worked?
Not on its own. Take the leavers out of last year's figure first. If the people who were there both times moved, something changed for them. If they did not, the rise is turnover, not improvement.
Do new starters affect the score too?
Yes. New starters often score higher in their first year, so hiring lifts the headline as well. Put them in separately and the calculator removes their effect alongside the leavers'.
Cite this tool
Free to use and free to quote. If a figure here is useful in something you are writing, a link back is all that is asked.
Muhammad Shahzeb Anwar, MSc Industrial and Organizational Psychology. Survivorship Bias Calculator. Shahzeb Speaks. https://www.shahzebspeaks.com/survivorship-bias-calculator/
The other free tools
Each one answers a single question and refuses the ones it cannot. All free, none asks for an email address.
Want the shape of the answers, not just the score?
The paid Employee Survey Results Analyser reads what this page leaves alone: whether a team averaging three is a team in the middle or a team split between the two ends, which need opposite responses. This page asks who is missing from the results. That one reads the ones that are there.
See the paid tool