I’ve spent years watching education leaders make decisions based on numbers that sound reasonable until you look closer. A district reports that test scores improved by 3 points on average. A school celebrates that 85% of students met the standard. A program shows a positive effect size of 0.4. These figures get presented in reports, defended in meetings, and used to justify continuing or scaling interventions. But they often obscure the actual state of things.
The problem isn’t that averages are useless. It’s that they answer only one question: what is the typical outcome? They don’t answer the questions that matter most for policy: Who is still struggling? How wide is the gap between the highest and lowest performers? Are we moving the needle for the students who need it most, or just for those already doing well?
I’ve seen this play out repeatedly. A reading intervention shows a mean gain of 0.5 grade levels across the cohort. That looks promising. But when you disaggregate the data, half the students gained nothing measurable, while the other half made substantial progress. The average masked a bimodal distribution. The policy question then becomes: why did the intervention work for some students but not others? That’s not answerable from the mean alone.
The Distribution Problem
When you work with classroom data long enough, you start to see that distributions matter far more than single summary statistics. Two classrooms can have identical average scores and look completely different when you plot the actual performance of each student.
One class might have most students clustered tightly around the mean, with few outliers. Another might have a wide spread – some students performing well above grade level, others significantly below, with fewer students in the middle. Same average. Completely different teaching problem. The first class probably needs refinement and enrichment. The second needs differentiation and targeted support.
Policy makers who rely on averages often miss this entirely. They see a district average and assume the intervention is working uniformly. They don’t see the school where performance is bimodal, or the subgroup where outcomes are actually declining while the overall average stays flat. These patterns only emerge when you look at the actual distribution of outcomes, not the summary statistic.
I’ve watched schools implement a new math curriculum based on average gains, only to discover later that struggling students fell further behind while advanced students pulled ahead. The average improved because the gains at the top outweighed the losses at the bottom. That’s not success. That’s widening inequality with a positive mean.
Variation as Signal, Not Noise
There’s a tendency in policy work to treat variation as something to smooth over or ignore. High variation in outcomes looks messy. Low variation looks clean and controlled. But variation often tells you something important about what’s actually happening in schools.
Large variation in student outcomes within a school or classroom sometimes signals that instruction isn’t consistent, or that some teachers are reaching students while others aren’t. It can indicate that some students have access to resources or support that others lack. It can reveal that the policy or program isn’t being implemented uniformly. These are all things worth knowing. They’re not problems to hide behind an average.
Conversely, artificially low variation can be a warning sign. If every student in a class performs at nearly the same level, it might mean the instruction is well-targeted and differentiated. Or it might mean that struggling students have been removed from the class, or that the assessment isn’t sensitive enough to capture real differences in understanding. You can’t tell from the average alone.
The Floor and Ceiling Matter
I’ve noticed that policy conversations almost never focus on the students at the extremes, yet those students often reveal the most about whether a policy is actually working. What’s happening with the lowest-performing students? Are they making progress, staying flat, or falling further behind? What about the highest performers – are they being challenged, or are they coasting?
A policy that raises the average while leaving the lowest performers behind isn’t closing gaps. It’s widening them. Yet this happens regularly because averages can mask it. A school might show a 2-point average gain, but if that gain came entirely from students already performing above the median, the students who need the most help are no better off.
Similarly, I’ve seen gifted programs justified by average gains that actually reflect only the performance of advanced students. The policy looks successful on paper because the mean is high. But if you disaggregate by prior achievement level, you find that students entering the program at lower levels made minimal progress. The average was inflated by students who would have performed well regardless.
Subgroup Performance as Essential Context
Disaggregation by demographic subgroup, prior achievement level, or other meaningful categories isn’t an add-on to policy analysis. It’s fundamental. An intervention that works on average but fails for a specific subgroup isn’t a successful intervention. It’s an inequitable one.
I’ve reviewed countless program evaluations that report a positive overall effect but show no effect or negative effects for particular groups. Sometimes this is presented as a limitation. Often it’s buried in an appendix or mentioned in passing. But it should be front and center in any policy decision. If a reading program works well for students with strong oral language skills but not for English learners, that’s not a minor detail. That’s a fundamental finding about who the program serves and who it fails.
The same applies to prior achievement. A tutoring program might show gains on average, but if those gains are concentrated among students who were already performing near grade level, while students performing well below grade level show minimal progress, the program isn’t addressing the core need. The policy question shifts from “does this work?” to “for whom does this work, and why?”
Disaggregated data also reveals something about implementation quality. If one school shows strong results and another shows weak results with the same intervention, that difference tells you something about how the program is being delivered, the quality of staff training, or the school context. The average masks these differences. The disaggregated view exposes them.
When Averages Can Mislead About Progress
Year-to-year average gains can be particularly deceptive. A district shows that math scores improved by 2 points this year compared to last year. That sounds like progress. But if you look at the cohort data – the same students tracked over time – you might find something different. Maybe the improvement came because a new cohort of higher-performing students entered the grade level, while the previous cohort of lower performers moved on. The average improved, but individual student progress was flat or negative.
This is why value-added models, despite their limitations, try to track individual student growth rather than relying on cross-sectional averages. They’re attempting to answer a different question: did students actually learn more this year than they would have without this intervention? That’s harder to see in aggregate data.
I’ve also seen situations where average gains hide declining performance in a critical subgroup. A school’s overall reading scores go up, but performance for students with learning disabilities stays flat while performance for general education students improves. The average went up. Equity went down. A policy maker looking only at the average would declare success.
The Practical Implication for Policy Design
Effective policy requires looking at distributions, not just central tendencies. It requires disaggregating data by meaningful groups. It requires asking not just “what is the average outcome?” but “what is the range of outcomes, and who falls where?” It requires tracking individual progress over time, not just comparing cohorts year to year.
This doesn’t mean averages are useless. They’re useful for understanding the typical case. But they’re insufficient for understanding whether a policy is actually working for the students who need it most. A policy that improves the average while leaving the lowest performers behind is a policy that increases inequality. A policy that narrows the distribution and raises the floor is a policy worth scaling.
The hard part is that this kind of analysis takes more work than reporting a single number. It requires disaggregation, visualization, and careful interpretation of patterns. It requires acknowledging complexity rather than reducing it to a headline. But that complexity is where the actual policy question lives. Ignoring it doesn’t make decisions easier. It makes them less informed.
