Ask Jev the same thing twice

If the exact same request is sent twice, how much does Jev's answer change, and when does its top answer flip?

result198,609 questions

Send Jev the exact same request twice and its probability moves by 1.3 points on average: 2.5 points near 50/50, under 1 point when it's more than 20 points from 50/50. Its top answer changes 1.7% of the time, and 99% of those flips happen within 10 points of a coin toss. Beyond 20 points from 50/50, it essentially never flips (1 in 147,652).

How to read this: Each dot is a band of how far Jev's first answer was from 50/50, closest on the left; its height is how often the favored option flipped on the repeat. Near coin tosses flip; anything decided doesn't.

198,609 pairs of identical requests; 95th percentile change 5 points, largest 40. The same noise shows up between the base probe and the same-order shuffle probe (1.3 points), which differ only in when they were sent.

In short

Asked the identical question twice, Jev's probability shifts by 1.3 points on average, and 95% of repeats move 5 points or less.
Its favored answer flips 1.7% of the time, almost always within 10 points of 50/50; beyond 20 points it flipped once in 147,652 pairs.
So a gap of a few points between two of Jev's answers is noise, and anything beyond about 5 points is real.

What the data shows

Say it Again, Dexter meme: asks Jev a question; asks it again, word for word

Average wobble: 1.3 points. 95% of repeats move 5 points or less; the largest move was 40.
Flips happen only at toss-ups: 39% of pairs within 2 points of 50/50 flip, 16% at 2 to 5 points, 2% at 5 to 10, 0.1% at 10 to 20, and 1 in 147,652 beyond 20.
The wobble shrinks when Jev is sure: 2.5 points near 50/50, 0.9 points when it's more than 20 points away.

What it means, and what it doesn't

Jev's answers are stable. A difference of a few points between two answers is noise; anything beyond about 5 points, or any answer that isn't a near toss-up, is real. That's the noise floor every other experiment on the site can be read against.

It doesn't mean Jev knows the answer: a near toss-up like Japan versus Liberia stays a near toss-up however many times you ask.

Caveats

Rounded probabilities. Jev's probabilities come back rounded to whole points, so changes under a point are invisible, and a split a point either side of even can flip on rounding alone.
Two-option questions only. The repeated requests exist only for two-option questions, so the noise on longer lists and ratings is assumed, not measured, to be similar.

Jev on this experiment

Would a person find it interesting to read?: Yes64%
Does it describe you?: Yes52%
Would you have predicted it?: No51%
How fair is the comparison?: The comparison is fair
How much should a reader rely on it?: Moderately
Which caveat matters most?: Rounded probabilities94%

Why ask this

Every comparison on this site rests on a question: how much would Jev's answer change if you just asked again? Many chatbots answer differently each time. Jev returns probabilities rather than a sampled answer, so the question is whether those probabilities are fixed, or wobble, and whether a wobble can change what it would pick.

It matters beyond this site too. A spam filter that says "spam" today and "fine" tomorrow on the same email, or a review check that flips on a resubmitted form, is hard to trust, and hard to debug.

How this was done

The people and the data

No people; Jev against itself. Each two-option question on the site is asked three times as part of its usual checks: with the options reversed, in the original order, and reversed again. The two reversed requests are identical word for word and were sent separately, at different times. That gives 198,609 pairs of identical requests to compare.

What Jev was asked

Any two-option question, twice. For example:

Which country is larger by area: Liberia or Japan?
Japan · Liberia

(Jev leaned only slightly toward Japan: right, but a near toss-up on a question it should have been sure of.)

How it was measured

The absolute change in the probability of the same option between the two identical requests, and the share of pairs where the favored option switched, grouped by how far the first answer was from 50/50.

Where these questions live

198,609 questions across 1,080 topics of the map; the 16 biggest are shown. Each opens on the map with every question in it.

Every question

All 198,609 questions behind this result, the telling ones first: the examples the analysis points to, then the ones where Jev misses, biggest gap first.

Jev’s own answer

Showing 0 of 198,609