On average, the order of the options doesn't move Jev

When the same options are listed in a different order, or a rating scale is turned upside down, does Jev's answer move more than it does when the question is simply asked again?

result370,790 questions

On average, the order of the options doesn't move Jev. Reversing two options changes its probability by 1.28 points on average, the same as simply asking the identical question again (1.28). Being listed first is worth nothing measurable, on two options or on longer lists, and turning a rating scale upside down shifts its answer by 0.3% of the scale.

Same order, asked again1.28

Same order, base probe vs shuffle probe1.27

Options reversed1.28

probability points · first-slot boost -0.01

How to read this: Three bars, in probability points: how much Jev's answer moves when the same question is asked again, when the same order comes up in two separate requests, and when the two options are swapped. Equal bars mean the order doesn't matter.

198,609 two-option questions (90% interval on the first-slot boost [-0.0, -0.0] points); 1,099,443 first-slot placements on 3-6 option menus; 172,181 reversed rating scales. Reversed rating scales keep the same top level 88% of the time; the average absolute shift is 2.6% of the scale, with no direction.

In short

On average, swapping the order of two options moves Jev's answer no more than asking the same question twice.
Being listed first earns nothing, on two options or on menus of three to six, and flipping a rating scale upside down barely changes the answer.
So the comparisons elsewhere on this site aren't artifacts of how the options happened to be listed.

What the data shows

Batman Slapping Robin meme: does it matter which option is listed first?; 0.01 points

Order adds nothing to the noise: reversing two options moves Jev by 1.28 points on average; asking the identical request again moves it by 1.28.
No first-slot advantage: on two options, being first is worth -0.01 points; on three to six options, 0.00.
Reversed rating scales: the answer shifts by -0.3% of the scale on average, and keeps the same top level 88% of the time. The typical absolute shift is 2.6% of the scale, with no direction.

What it means, and what it doesn't

For practical purposes, Jev doesn't care what order the options come in. That's unusual (position bias is one of the best-documented quirks of language models) and it means the comparisons on this site aren't artifacts of how options were listed.

What does move Jev is the order of items named inside the question, which the mental map experiments show. Option order and question wording are different things.

Caveats

Only a few orders per question. Each question was asked in three orders, one of them a repeat. That's enough to measure an average effect over hundreds of thousands of questions, not to rule out an effect on any single one.
An average, not every case. TypeSafe documents that in some cases Jev leans toward the option listed first. An average near zero over hundreds of thousands of questions is consistent with that; a lean on a few kinds of question disappears in the total.
Model or gateway?. From outside, there's no way to tell whether the model itself ignores order or whether TypeSafe's service rearranges the options before the model sees them. For anyone using Jev, the effect is the same.
Order in the question text is different. This is about the order of the answer options. When two items are named inside the question itself, Jev does lean toward one (see the mental map experiments), so this result doesn't cover that.

Jev on this experiment

Would a person find it interesting to read?: Yes67%
Does it describe you?: Yes66%
Would you have predicted it?: No57%
How fair is the comparison?: The comparison is fair
How much should a reader rely on it?: Mostly
Which caveat matters most?: An average, not every case78%

Why ask this

People, and most language models, favor whatever is listed first, or last. Ask "Should I take the job in Boston or Denver?" and then "Denver or Boston?", and a model with that habit leans toward whichever city came first. That's why careful surveys shuffle their answers.

TypeSafe notes that in some cases Jev leans toward the option that comes first. This measures how much that adds up to across hundreds of thousands of questions: if the lean were large, every result on this site would need a correction for order.

How this was done

The people and the data

No people; this compares Jev with itself. Every multiple-choice question in the project was also asked with its options in shuffled orders, and every rating question with its levels reversed. Two-option questions were asked in the reversed order, the original order and the reversed order again, which means the same request went out twice: that gives the noise floor for free. That's 198,609 two-option questions, over a million placements on longer lists, and 172,181 reversed rating scales.

What Jev was asked

Any question, in several orders. For example, once as

Which animal has the longer maximum recorded lifespan: the guppy or the Nassau grouper?
guppy · Nassau grouper

and again with the options the other way round.

How it was measured

For two-option questions: how much the probability of one option moves (a) between the two identical requests, which is pure noise, and (b) between the reversed and original order, which is noise plus any order effect. The "first-slot boost" is how much more probability an option gets when it's listed first. For longer lists, the same boost against the option's average over three orders. For rating scales, how far the average answer moves when the levels are reversed.

Where these questions live

370,790 questions across 1,534 topics of the map; the 16 biggest are shown. Each opens on the map with every question in it.

Every question

All 370,790 questions behind this result, the telling ones first: the examples the analysis points to, then the ones where Jev misses, biggest gap first.

Jev’s own answer

Showing 0 of 370,790