NPR drops dumb Bell Curve segment

This morning, National Public Radio’s Morning Edition featured a segment entitled “Put Away The Bell Curve: Most Of Us Aren’t ‘Average’“. I am generally vigilant about stories which make broad claims about human traits and their genetic and environmental underpinnings, and this particular segment triggered my alarms to scream. Analyzing a new study on “academics writing papers, athletes at the professional and collegiate levels, politicians and entertainers”, the NPR segment more-than-just suggested that the bell curve representation of our abilities may be wrong. Given the history of the bell curve as a political tool, perhaps the editors at NPR thought this might be a nice counter to conventional wisdom. But how they interpret this study is so off base that it had me retreating to the safety of the bell curve.

The main thrust of the story is that we ought to reconsider whether the bell curve’s representation of human attributes accurately represents the distribution of these traits in real societies. If our “performance” is best represented by the bell curve, that suggests that most of us are average: in statistical terms, the mode and the mean are the same value. What this new study suggests — based on the metrics of performance it chose and the arenas in which it chose to look at this performance — is that most of us are below average, because statistics show that that our “performance” is not well-represented by the bell curve. The figure below, featured in the paper, gives a sense of this difference:

Image source: Personnel Psychology Volume 65, Issue 1, pages 79-119, 27 FEB 2012

In such a figure the horizontal axis represents the value of a particular measured trait and the vertical axis represents the frequency of that trait in the measured population. The bell curve is shown by the black line, and the defining feature of the bell curve is that the average trait value (the “mean”) is also the most frequent trait value (the “mode”), thus the symmetrical “peaked” shape of the line representing the distribution of trait values. Other distributions of traits are not symmetrical, and we call this asymmetry “skew”. The dark gray curve on the diagram above is left-skewed, meaning that the most common (modal) trait value is much lower than the average (mean) trait value.

The authors of this study present the following data in support of the idea that “performance” is not distributed as a bell curve:

Image source: Personnel Psychology Volume 65, Issue 1, pages 79-119, 27 FEB 2012

Based on their samples, there is pretty convincing evidence that populations of Researchers, Emmy-nominated actors, U.S. Representatives, NBA players, and MLB players do not show performance scores that are well-represented by a bell curve.

But does this mean that we as a society are not well-represented by the bell curve?

To answer this question, let us first consider the population of people whose “performance” was considered. All five of these populations are in some sense elite. A tiny fraction of the population gets to become a researcher, or an actor nominated for an Emmy, or a member of any of these populations. In fact, the percentage of the total population who are represented in this sample is infinitesimal, but this is not the only problem: each of these are also highly selected populations. The NBA draft is not a population-wide lottery (imagine what it would be like if it were!), it is a highly-selective process that culls the very best basketball players from the general population. So immediately we need to recognize that the results of this study cannot be applied to the population as a whole, because the trends discovered only apply to the select subpopulations considered.

So what does this have to say about whether or not our “performance” is well-represented by the bell curve? Well, perhaps we should consider what relation the chosen subsamples of these five populations have to the rest of society. Is it possible that our various talents are in fact well represented by the bell curve? As the graphic below suggests, the answer is clearly “yes”:

Image modified based on an original image by Petter Strandmark

If you only consider the top performers in any given arena, of course their distribution will be skewed to the left. The most elite performers on the right-hand side of the bell curve inevitably show a skewed distribution. This is an expected and unremarkable result of the very stringent selective processes that determine who can play Major League Baseball and who can get elected to Congress. These results have nothing to say about the distribution of “performance” across the whole society. Imagine if the NBA did draft people randomly from the population as a whole: you can bet that any performance statistic that you pulled would be distributed more closely to the bell curve than the current NBA. Obvious stuff, really.

The other big objection that I have to this study’s portrayal in the NPR piece has to do with the implicit connection the segment makes between our traits as a population and our performance. The bell or normal curve frequently provides a good representation of the distribution of traits, not performance, in a population. While sometimes differences in traits can lead to differences in performance, the link between the two can be pretty indirect, and this is a reality that the NPR piece totally glosses over. The article has some pretty big genetic determinist overtones, as “we” as people are treated as one and the same as our “performance”. But does a study of performance by a particular criteria really tell us much about the distribution of talent in our society?

“Performance” comes as much from training as from inborn talent; in fact, it is the combination of the two — not one or the other — that determines performance. What makes someone good at something always has to do with the experience and training they are privileged to have, and what is really skewed in our society is this privilege: a few people have a lot of privilege and most have very little. I am not just referring here to the massive economic and educational inequities that permeate our society, I am also referring to much more specific factors relating to the populations included in this study:

1. Researchers who publish good papers early in their career garner far more support for their work than those who publish less, and more support leads to further publication. This positive feedback loop can amplify rather small differences in the initial performance of these researchers, providing us with a distorted idea of what determines “performance” in this area.
2. Emmy nominations undoubtedly are impacted by past performances: if you have been nominated before, you are certainly more likely to be nominated for the same quality of performance than a person who has never been nominated. Again, it is easy to see how slight differences in initial talent might be amplified by our reward system.
3. This one seems almost too obvious to explain, but I will do it anyway. Members of Congress who manage to hold onto several terms (perhaps by very marginal differences in their job performance, or even by luck) climb in seniority on critical committees and then suddenly become much better supported by donors who wish to win influence. Few will get this level of support, but once they get it their re-election becomes far more likely than “average” representatives.

Perhaps the objections for the three populations above do not apply as well to the two populations of elite athletes sampled in the study, as in these occupations performance metrics are a direct rather than indirect measure of ability/talent. But still we should acknowledge that athletes who show marginal differences in initial ability or performance are going to be provided with the best training opportunities: if nothing else, the better players play against each other. We live in a society that is often “winner takes all”, and I think that all this study does is show that the elite among us contain subsets of the even more elite. Most of us may, in fact, be pretty average after all — we just live in a society that does not produce average outcomes. This failure to recognize the difference between potential ability and performance outcomes is in my eyes the most frustrating distortion provided by this NPR segment.

The researchers who produced this work seem to have their own reasons for conducting the study. In the paper they express the need to re-assess whether models of institutional performance should assume a bell-curved distribution of worker ability. This is reasonable given that most of the institutions they are imagining have some of the same selective processes as their sample: just as not everyone can be a player in the NBA, not everyone can get a job in a corporate office. But even when applied in this manner I worry that their sample data includes populations that are way too extreme in their performance to be the norm. Sure, we do not have every person in the population represented in the corporate office, but this is also not as selective a group as the U.S. Congress or even academic researchers. If I were in the business of managing a big corporation, I would be wary of applying these results based on extreme populations to my more average population of workers.

To summarize my worries about the effect of this piece, consider that listeners to the NPR story might walk away with the following wrong ideas:

1. We can draw conclusions about “we” as a broader population by looking at the performance of elite, select populations;
2. Looking at the outcomes produced by highly-trained and greatly-experienced individuals can tell us something about our own potential for performance; and
3. Particular talents can be generalized to understand the aptitude of whole individuals.

The bell curve has a really bad political history: bell curves based on very particular metrics for particular traits have been used to draw broad generalizations about the talents of our population. Beyond the biases that often are inherent in the chosen metrics, these statistical assessments are problematic because they can only capture one metric at a time, and I assure you that high “performance” on one metric is no guarantee of superior performance on all metrics. Our society is in part successful because some people are really good at some tasks and other are really good at others. One of the things we do as a society to take full advantage of this diversity is to aim additional opportunity and training at those who show a particular talent, thereby amplifying differences in initial talent. Are we all below average in performance? Sure, on the 99% of things we did not train to do. But we all have strong talents as well, talents developed by our life experiences. Our various talents and aptitudes probably do map out on thousands of different bell curves, but who cares?: to succeed one need only excel at a few things. NPR would have you look at the best among us in a particular area and hang our heads in shame at being so generally below average.

I really wonder whether the researchers who conducted this study endorse the conclusions made by this incredibly misleading and just-plain-wrong story. I am flabbergasted that this story got past NPR‘s editors; whoever was on duty clearly has little understanding of what the bell curve represents for biologists. Shame on NPR for this profoundly ignorant piece.