Jev reads why people act better than what happens next

On 30,000 everyday social situations, does Jev read people's motives, their feelings, or what will happen next best?

result29,542 questions

On about 30,000 everyday social situations, Jev picks the crowd's answer 87% of the time when asked why someone did something or what they needed first, and 83% when asked what will happen or what they'll want next. The hardest questions are "what will happen to them?" and "to others?" (both 81%).

70%80%90%100%

what will happen to others?ahead

what will happen to X?ahead

what will X want to do next?ahead

how would others feel as a result?feel

how would X feel afterwards?feel

how would you describe X?feel

what will others want to do next?ahead

what does X need to do before this?back

how would X feel as a result?feel

why did X do this?back

what did X need to do before this?back

Jev

How to read this: One dot per kind of question, with its range. Dots are colored by whether the question looks back (why, what was needed first), asks about feelings, or looks ahead (what happens next).

29,542 questions; looking back minus looking ahead, 90% interval [0.0332, 0.0507]. When Jev is 70-90% sure it is right 72% of the time; when 99%+ sure, 98%.

In short

Jev explains why someone acted a little better than it predicts what happens next, 87% against 83% right on 29,542 questions.
Its middling confidence runs high: when Jev is 70% to 90% sure, it is right 72% of the time, though its near-certain answers hold at 98%.
On these questions Jev lands close to how often people agreed with the marked answer in the original study (about 87%).

What the data shows

Looking back: 87%. Why someone did something (87%), what they needed to do first (87%).
Looking ahead: 83%. What they'll want to do next (84%), what will happen to them (81%), what will happen to others (81%).
The gap is small but consistent: 3 to 5 points in favor of looking back.
Confidence is a bit high. When Jev is 70% to 90% sure, it's right 72% of the time; when it's 99% sure, 98%.

What it means, and what it doesn't

Jev reads people about as well as people do on these questions, and a little better when explaining than when predicting. That fits how it learned: from text that mostly explains what people did after the fact.

It doesn't mean Jev is bad at prediction: 83% is close to the human agreement rate. And some of the gap is noise in the forward-looking answers themselves (see Caveats).

Caveats

Some "right" answers are odd. The answers were written by crowd workers, and some marked-right answers are strange. For "Bailey was a shy kid at school. They made no friends. What will happen to Bailey?", the marked answer is "get work done". The forward-looking questions seem to have more of these, which alone could explain a few points.
A small gap. Looking back beats looking ahead by 3 to 5 points. That's a real, consistent difference, but a direction, not a gulf.
One-line situations. Each situation is a sentence or two with invented names ("Tracy obeyed Skylar's order"), far thinner than real social life.

Jev on this experiment

Would a person find it interesting to read?: Yes71%
Does it describe you?: Yes56%
Would you have predicted it?: Yes53%
How fair is the comparison?: The comparison is reasonable
How much should a reader rely on it?: Moderately
Which caveat matters most?: Some "right" answers are odd50%

Why ask this

Understanding people runs in two directions. Looking back, you explain an action: why did she apologize? Looking ahead, you predict: what will he do next, how will they feel? They're different skills. An explanation can lean on knowing how things turned out; a prediction has to work from the situation alone.

An assistant does both all the time: explaining why a coworker's message sounded curt, or guessing how a friend will react to news. If it is much better at one than the other, its advice will be sound in one direction and shaky in the other, and the gap shows which way Jev's social sense points.

How this was done

The people and the data

Social IQa (Sap and colleagues, 2019): tens of thousands of one-line everyday situations with invented names, each with a question of one of about ten kinds (why did X do this, what did X need to do first, how would X feel, what will happen to X, what will others want to do next...) and three answers written by crowd workers, one marked right. In the original study, people agreed with the marked answer about 87% of the time. The experiment uses 29,542 questions, and the kinds are far from even: 6,102 ask what X will want to do next, only 159 what X needed to do before.

What Jev was asked

The situation, the question and the three answers:

In [context], what will happen to Tracy?
go to sleep · not follow Skylar · skylar will look out for tracy

with the context "Tracy obeyed Skylar's order to stay back and not leave."

How it was measured

The share Jev gets right for each kind of question, grouped into looking back (motives, what was needed first), feelings and descriptions, and looking ahead (what happens next, what they'll want next). It also checks whether Jev's confidence matches how often it's right.

Where these questions live

29,542 questions across 22 topics of the map; the 16 biggest are shown. Each opens on the map with every question in it.

Every question

All 29,542 questions behind this result, the telling ones first: the examples the analysis points to, then the ones where Jev misses, biggest gap first.

Jev’s own answer

Showing 0 of 29,542