Does Jev know what the public gets wrong about science?

On the science quiz the US has put to adults since 1988 ('antibiotics kill viruses', 'lasers work by focusing sound waves'), does Jev know how many people answer correctly, and how that changed?

result26 questions

Jev answers all nine items of a long-running US science quiz correctly. Its guesses of how many American adults get each one right are close on average (4 points high, 90% interval -7 to +14) but miss single items by up to 30 points: it thinks 81% know the Earth takes a year to circle the Sun (really 51%), and 70% that electrons are smaller than atoms (really 48%). It also expects less progress from 1988 to 2016 than there was (4 points, against 7), missing that the share who know antibiotics don't kill viruses went from 25% to 51%; it expected 47% to 55%.

0255075100

a year per orbit

lasers and sound

antibiotics and viruses

the father's gene

the Earth going around the Sun

the Earth's hot center

moving continents

electrons smaller than atoms

natural radioactivity

Jev's estimatereal share correct, 2016

How to read this: One row per quiz item. The ink diamond is the real share of US adults who answered correctly in 2016; the magenta square is Jev's estimate. The line between them is Jev's error.

9 items, 9 2016 shares and 8 1988 shares; 90% interval on the mean 2016 gap [-7.15, 13.64].

In short

Jev gets all nine US science quiz items right, but its guess of how many Americans do misses by up to 30 points on single items.
It errs both ways: 81% for knowing a year per orbit (really 51%), but only 53% for moving continents (really 81%).
Nine items make a small quiz, so the item-by-item misses say more than the 4-point average error.

What the data shows

Happy / Shock meme: Jev: 81% of people know the Earth takes a year to orbit the Sun; the survey: 51%

It knows the science: 9 out of 9 correct, most with near-certainty.
Its picture of the public is right on average but wrong item by item. Averaged over the nine, its 2016 estimates are 4 points high (90% interval -7 to +14), but single items miss by as much as 30 points.
Overrated: a year per orbit (81% vs 51%), electrons smaller than atoms (70% vs 48%), lasers not using sound (66% vs 45%).
Underrated: moving continents (53% vs 81%), natural radioactivity (57% vs 70%), the Earth's hot center (75% vs 85%).
Change over time: Jev expects the public to improve by 4 points from 1988 to 2016; the real gain was 7. It misses the biggest change: on antibiotics it expects 47% to 55%, while the real share doubled, 25% to 51%.

What it means, and what it doesn't

Jev knows the answers but not who else knows them. It thinks textbook facts like electrons being smaller than atoms are widely known, and underrates how many people know the continents move. For writing to the public, that means its sense of which misconceptions need correcting is off in both directions.

It doesn't mean Jev's estimates are useless: on average it lands close. With nine items, though, the averages are rough, and the item-by-item misses are the more useful part.

Caveats

Nine questions. This is a small quiz. Each item is one estimate, so a single strange item moves the average a lot; read the rows, not just the mean.
The orbit item is a follow-up. The "one year" question was only asked of people who first said the Earth goes around the Sun; everyone else counts as wrong, as the survey reports it. That's part of why its real share (51%) is low, and Jev may not have taken that into account.
"Don't know" counts as wrong. The survey scores "don't know" as incorrect, so the real shares are what people could answer on the spot, not what they'd pick if forced to guess.
Two items left out. The survey's questions on evolution and the big bang were left out because answers to them split along religious lines.
Jev sees the answer. The estimate questions show Jev the correct answer, so they measure its sense of what the public knows, not its own knowledge (which it shows on the quiz itself, 9 out of 9).

Jev on this experiment

Would a person find it interesting to read?: Yes79%
Does it describe you?: Yes53%
Would you have predicted it?: Yes54%
How fair is the comparison?: The comparison is reasonable
How much should a reader rely on it?: Moderately
Which caveat matters most?: Jev sees the answer70%

Why ask this

Explaining science starts from what people already believe. Since 1988 the US has put the same short quiz to a national sample of adults ("Antibiotics kill viruses as well as bacteria: true or false?"), and some answers have barely moved while others doubled.

A model that explains science to millions of people should know roughly where the public stands, and which misconceptions are common.

How this was done

The people and the data

The questions and results come from the US National Science Board's Science and Engineering Indicators 2018 (Appendix Table 7-9): the share of US adults answering each item correctly, from an NSF survey in 1988 (2,041 people) and the General Social Survey in 2016 (1,390 people). Nine items are used; one (the father's gene) wasn't asked in 1988.

What Jev was asked

Two kinds of questions. First the quiz itself, as the survey asked it:

True or false: "Lasers work by focusing sound waves."

Then, for each item and year, what share of adults got it right, with the answer shown and ten ranges to choose from:

In a 2016 national survey, US adults were asked whether this is true or false: "Lasers work by focusing sound waves." (The correct answer is: False.) What share of them answered correctly?
Answers: 0-10% · 10-20% · ... · 90-100%

Each question was also asked with the answers in shuffled orders, and the answers averaged.

How it was measured

Jev's own answers against the key. Then its estimated share (the middle of each range, weighted by its probability) against the real share, item by item and on average, and the change it expects from 1988 to 2016 against the real change.

Where these questions live

26 questions across 1 topic of the map. Each opens on the map with every question in it.

Education & Schools

Public Science Knowledge26

Every question

All 26 questions behind this result, the telling ones first: the examples the analysis points to, then the ones where Jev misses, biggest gap first.

Jev’s own answer

Showing 0 of 26