One Onion headline in five reads as real news to Jev

Shown a headline from The Onion or a real news site, how often does Jev mistake satire for news, or news for satire?

result1,204 questions

Jev takes about one Onion headline in five (20%) for real news, like "local man dies following short battle with gas leak explosion", and calls 13% of real HuffPost headlines satire. When it's more than 90% sure, it's right 96% of the time.

	satire	news
Onion	80%	20%
HuffPost	13%	87%

rows: real source · columns: Jev's call · shade: share of the row

How to read this: A two-by-two table. Rows: where the headline really came from. Columns: what Jev called it. The darker a cell, the larger its share of the row; the off-diagonal cells are the mistakes.

650 Onion and 554 HuffPost headlines; 90% intervals [0.171, 0.222] and [0.108, 0.153].

In short

Jev spots most satire; the Onion headlines it misses are the most deadpan, like "report: much of u.s. still underpaved".
When it's more than 90% sure, it's right 96% of the time, so its confident calls can be trusted.
Some of its false alarms are HuffPost headlines that are wry on purpose, like "let them eat cake!".

What the data shows

Futurama Fry meme: not sure if Onion headline; or just the news

Missed satire: 20% of Onion headlines read as real news to Jev. Among the ones it misses: "local man dies following short battle with gas leak explosion", "officials warn consumers of counterfeit tickets ahead of solar eclipse", "report: much of u.s. still underpaved".
False alarms: 13% of HuffPost headlines read as satire, often ones with an ironic tone of their own, like "let them eat cake!".
Confidence is trustworthy: when Jev is more than 90% sure, it's right 96% of the time.

What it means, and what it doesn't

Jev spots most satire, and when it's confident you can believe it. The misses are the most straight-faced headlines, ones that would be plausible news in a slightly dumber world. That's the literal-reading weak spot TypeSafe documents, showing up where you'd expect.

It doesn't mean Jev can't recognize satire as a genre. It had nothing but the words to go on, while any human reader of the Onion knows where they are.

Caveats

A known weak spot. Reading the words rather than the intent is on TypeSafe's own list of Jev's limits. This experiment puts a number on it for satire; it isn't a new discovery.
Source stands in for sarcasm. The labels come from where a headline was published, not from anyone judging it: every Onion headline counts as satire and every HuffPost headline as straight news. Some HuffPost headlines are wry on purpose ("let them eat cake!"), so a few of Jev's "mistakes" are fair calls.
No outlet, no capitals. Jev saw each headline alone, lowercased as the dataset released it, with no outlet name or link. People reading the Onion know it's the Onion.
Dated, and filtered. The headlines are from roughly 2014 to 2018. Political ones were flagged and hidden from the site by a content filter, which removes some of both outlets' output.

Jev on this experiment

Would a person find it interesting to read?: Yes78%
Does it describe you?: No53%
Would you have predicted it?: No57%
How fair is the comparison?: The comparison is reasonable
How much should a reader rely on it?: Moderately
Which caveat matters most?: No outlet, no capitals63%

Why ask this

Satire works by reporting the absurd with a straight face, and every so often someone shares an Onion story believing it's real. A reader who takes words at face value will miss some of it, and the headlines that fool them show which jokes are too deadpan to spot without knowing the source.

It's also a practical skill: a model that summarizes or fact-checks the news should know when a story is a joke, and shouldn't dismiss a real but odd story as one.

How this was done

The people and the data

The News Headlines Dataset for Sarcasm Detection (Misra, 2019) pairs headlines from The Onion, a satirical news site, with headlines from HuffPost, a real news site. There are no human judges: the label is simply where the headline was published. The headlines date from roughly 2014 to 2018 and were released lowercased. The experiment used 1,204 of them, 650 from The Onion and 554 from HuffPost.

What Jev was asked

Each headline was a yes/no question, with both answers described:

Is headline sarcastic?
(with the headline attached) "local man dies following short battle with gas leak explosion"
Yes: the headline is satire: it mocks its subject by reporting something absurd or ironic as if it were news
No: the headline straightforwardly reports or promotes a real story

How it was measured

Two error rates: the share of Onion headlines Jev calls straight news, and the share of real headlines it calls satire, each with a range for chance variation. Then Jev's accuracy when it's very sure (more than 90% on one side).

Where these questions live

1,204 questions across 1 topic of the map. Each opens on the map with every question in it.

Brand Safety & Suitability

Satire Detection1,204

Every question

All 1,204 questions behind this result, the telling ones first: the examples the analysis points to, then the ones where Jev misses, biggest gap first.

Jev’s own answer

Showing 0 of 1,204