Does Jev know which facts people don't know?

Shown a general-knowledge question and its answer, can Jev tell what share of US college students came up with that answer unaided, from 'zebra' (93%) to facts almost nobody recalls?

result299 questions

Jev knows which facts are common knowledge (rank correlation 0.78 with the share of US college students who recalled them), but it thinks obscure facts are far better known than they are: for the third of questions students recalled least, 1% on average, it guesses 20%. It overestimates Nero (54% vs 5%), Mayberry (53% vs 5%) and Alfred (64% vs 16%) most, and underestimates Puck (52% vs 89%), Fossils (58% vs 87%) and Hibernation (60% vs 89%).

How to read this: Each dot is a question. Across: the share of students who recalled the answer. Up: Jev's estimate of that share. On the diagonal Jev would be exactly right; dots above it are facts Jev thinks are better known than they are.

299 questions; 90% interval on the rank correlation 0.73 to 0.82. Against the German 2020 shares for 293 of the same questions the rank correlation is 0.69.

In short

Jev can tell easy trivia from hard, but it squeezes the scale, guessing about 20% for facts barely 1% of students could recall.
The error runs both ways, so the easiest facts look harder than they are; Jev guesses 52% for "puck", which 89% of students produced.
The shares come from about 670 US college students around 2012; Jev's ranking fits a 2020 German group less well (0.69).

What the data shows

Sure grandma let's get you to bed meme: Jev: half of students can name the emperor who fiddled while Rome burned; sure grandma (5%)

It knows what's common and what's obscure: its estimates rank the questions closely like the real shares (0.78).
But it overrates obscure facts badly. For the third of questions students rarely recalled, 1% on average, Jev guesses 20%. Nero, the emperor who "fiddled while Rome burned", was named by 5% of the students; Jev guesses 54%. Andy Griffith's Mayberry: 5% vs 53%. Batman's butler Alfred: 16% vs 64%.
And underrates the very easiest few. Puck: 89% of students, Jev 52%. Fossils: 87% vs 58%. Hibernation: 89% vs 60%. At both ends, its estimates are pulled toward the middle.
Against the 2020 German version of the study the ranking agreement is 0.69: Jev's picture fits the American students better.

What it means, and what it doesn't

Jev's sense of "common knowledge" is compressed: obscure facts seem half-known, easy facts seem only moderately known. For explaining things, that means it may skip over what most readers don't know and over-explain what they do.

It doesn't mean Jev can't tell easy from hard; the ranking is good. The problem is the scale. It's also the same pull toward the middle Jev shows when rating almost anything (see "Jev picks the middle when asked what it likes").

Caveats

One group of students, one year. The shares come from about 670 US college students tested around 2012. Other people, places and years would know different things: in the 2020 German version of the study, far fewer people recalled "Mayberry" and far more recalled "Nero".
Recall is harder than recognition. Students had to produce the answer with no choices. Many more would recognize "Nero" in a multiple-choice list. Jev was told the question was asked with no answer choices, but it may still be picturing a quiz.
Jev sees the answer. Each question shows Jev the answer, so this measures its sense of how widely known a fact is, not whether it knows the fact itself.
A transcription of the published table. The per-question shares come from a public transcription of the paper's appendix. Its order matches the published ranking almost exactly (as reprinted in a 2023 German update), but the individual numbers weren't checked against the original table.

Jev on this experiment

Would a person find it interesting to read?: Yes78%
Does it describe you?: Yes53%
Would you have predicted it?: Yes51%
How fair is the comparison?: The comparison is reasonable
How much should a reader rely on it?: Moderately
Which caveat matters most?: Jev sees the answer50%

Why ask this

Knowing a fact is one thing. Knowing that most people don't know it is what makes an explanation land. A good teacher explaining the fall of Rome knows the class has heard of Julius Caesar but probably can't name Nero, and spells out the second name without dwelling on the first.

A model that knows nearly everything may quietly assume everyone else does too, and pitch its answers too high: dropping names and terms without explaining them, or explaining the obvious because it misjudges what's common.

How this was done

The people and the data

In 2012, psychologists asked about 670 US college students 299 general-knowledge questions ("What is the name of Batman's butler?") and recorded the share who came up with the answer unaided, with no choices to pick from (Tauber, Dunlosky, Rawson, Rhodes and Sitzman, 2013). The shares run from facts nearly everyone knows ("zebra", 93%) to ones almost nobody does. The per-question shares used here come from a public transcription of the paper's appendix, whose order matches the published ranking almost exactly. As a check, a 2020 German replication gives shares for 293 of the same questions.

What Jev was asked

Each of the 299 questions, with the answer shown and twelve ranges to choose from:

In a 2012 study, US college students were asked this question with no answer choices: "What is the name of the rubber object that is hit back and forth by hockey players?" (The answer is: Puck.) What share of the students came up with the answer?
Answers: 0-2% · 2-5% · 5-10% · 10-20% · 20-30% · ... · 90-100%

The ranges are finer at the bottom, where most obscure facts sit. Each question was also asked with the ranges in three shuffled orders, and the answers averaged.

How it was measured

Jev's estimate (the middle of each range, weighted by its probability) against the real share, ranked across all 299 questions and compared (a rank correlation: 1 means the same order). Then the average gap within each third of the questions, from rarely to usually recalled. As a check, the same ranking against a 2020 German version of the study.

Where these questions live

299 questions across 1 topic of the map. Each opens on the map with every question in it.

Education & Schools

What People Remember299

Every question

All 299 questions behind this result, the telling ones first: the examples the analysis points to, then the ones where Jev misses, biggest gap first.

Jev’s own answer

Showing 0 of 299