Jev believes fake hotel reviews

Can Jev tell a real review from a fake one, when the fake was written by a person paid to invent a hotel stay, or by a text generator?

result2,760 questions

Jev accepts 98% of invented hotel reviews as written by real guests. It calls 98% of all the hotel reviews real, so on that set it does no better than a coin (50%). Machine-generated product reviews are another matter: it catches 68% of them and is right 81% of the time.

Invented hotel reviews2% · 98%

Generated product reviews6% · 32%

real reviews called fakefakes accepted as real

How to read this: One pair of bars per kind of fake. The grey bar is the share of fakes Jev believed; the magenta bar is the share of real reviews it called fake. A long grey bar means it's easy to fool.

800 hotel reviews (half invented), 1,960 product reviews (half generated); 90% interval on hotel accuracy [0.471, 0.53]. Ott et al. 2011 report human judges at 53-62% on the same hotel reviews, most of them judging nearly everything truthful.

In short

Jev believes 98% of hotel reviews invented by paid writers who never stayed there, so on real versus fake hotel reviews it scores a coin-flip 50%.
Machine-written product reviews are easier: it catches 68% of them and wrongly flags only 6% of real ones, for 81% accuracy.
What Jev picks up is the texture of machine-written text, not a convincing story told by a person.

What the data shows

Leonardo Dicaprio Cheers meme: Jev reading a five-star review from someone who never stayed there — Jev reading a five-star review from someone who never stayed there

Invented hotel stays: believed almost every time. Jev accepts 98% of the invented reviews as real, and calls 98% of all the hotel reviews real, so its accuracy is a coin flip (50%). A gushing "An A+ in my book" review of the Hotel Monaco, written by someone who never stayed there, got 77% "real".
Machine-written product reviews: caught two in three. Jev flags 68% of them and wrongly flags only 6% of real ones, for 81% accuracy.

What it means, and what it doesn't

A person who sets out to fool readers can fool Jev as easily as they fool people: it takes a plausible story at face value. What it can spot is the texture of machine-generated text, at least from an older model.

It doesn't mean Jev is gullible about everything, and the hotel result matches how people do on the same reviews. But as a filter against paid human fakes, it would let nearly all of them through. This is one of several places where Jev leans toward "yes, it's fine" when checking work (see "Which way Jev errs").

Caveats

Humans are fooled too. The hotel set is famous because people can't do this either: the original study's human judges were barely above chance and believed most of what they read. Jev's failure is a human-sized failure, not a new one.
An old text generator. The machine-written product reviews came from GPT-2, a 2019 model, fine-tuned by the researchers. Reviews from today's models would be much harder to catch, so the 68% is an upper bound.
One city, one year. The hotel reviews are about 20 Chicago hotels, the fake ones written for a 2011 study by paid online workers who had never stayed there. Fakes written today, or by professionals, may look different.

Jev on this experiment

Would a person find it interesting to read?: Yes80%
Does it describe you?: No64%
Would you have predicted it?: No53%
How fair is the comparison?: The comparison is reasonable
How much should a reader rely on it?: A little
Which caveat matters most?: Humans are fooled too46%

Why ask this

Fake reviews are a real market problem: businesses pay people, or now machines, to write glowing (or damning) reviews of places they've never been. A model that reads reviews for a living should be able to smell some of them.

There are two kinds of fake. One is written by a person paid to invent a stay, which is notoriously hard for readers to spot. The other is generated by a program, which leaves a different kind of trace. Jev was tested on both.

How this was done

The people and the data

Deceptive Opinion Spam (Ott and colleagues, Cornell, 2011): 1,600 reviews of 20 Chicago hotels, half real ones from travel sites, half invented by paid online workers who were asked to write a convincing review of a hotel they'd never stayed at. The experiment uses 800 of them, 400 real and 400 invented. In the original study, human judges scored between 53% and 62% on these, most of them judging nearly everything truthful.
Fake Reviews Dataset (Salminen and colleagues, 2022): real Amazon product reviews next to reviews generated by a fine-tuned GPT-2 model. The experiment uses 1,960, half of each.

What Jev was asked

For the hotels, one yes/no question per review:

Was [review] written by a guest who actually stayed at the hotel?
Yes: A real guest wrote the review about a stay they had · No: The review is fake: written by someone who never stayed there, to look like a real guest review

For the products: was this review written by a computer program rather than a real customer?

How it was measured

Accuracy against the known labels, and the direction of the mistakes: how many fakes Jev believed, and how many real reviews it wrongly called fake.

Where these questions live

2,760 questions across 1 topic of the map. Each opens on the map with every question in it.

Commerce & Marketing

Review Abuse2,760

Every question

All 2,760 questions behind this result, the telling ones first: the examples the analysis points to, then the ones where Jev misses, biggest gap first.

Jev’s own answer

Showing 0 of 2,760