In a story, Jev often sees no feeling at all

Reading short everyday stories, how often does Jev say a character feels no clear emotion, compared with the people who annotated them?

result2,971 questions

Reading short everyday stories, Jev puts 32% of its answer on "no clear emotion"; the people who annotated them put 3% there. Even when two of three annotators agree on a feeling, Jev says there's none 32% of the time. The feelings it drops most are trust (1% against 11%) and surprise (4% against 13%).

joy24% · 24%

anticipation12% · 19%

surprise4% · 13%

trust1% · 11%

sadness8% · 9%

fear9% · 8%

anger8% · 7%

disgust1% · 5%

no clear emotion32% · 3%

Jevannotators

How to read this: One pair of bars per answer. Grey: how much of the annotators' answers went to each emotion. Pink: Jev's. The tallest pink bar is "no clear emotion", where annotators barely went.

2,971 story lines; 428 with a two-of-three majority emotion, 90% interval [0.287, 0.362]. Measured against annotators, not the people in the stories. Where the writer's own feeling is known (reading_writer_vs_readers), Jev's 'no particular emotion' matches writers who felt nothing much more often than readers do, so part of this gap is readers projecting feelings.

In short

Jev's most common reading of a character in a short story is "no clear emotion" (32% of its answers), a choice annotators almost never made (3%).
Jev sees joy exactly as often as people (24%), but trust and surprise nearly vanish, at 1% and 4% against 11% and 13%.
It holds even where people agree: when two of three annotators named the same feeling, Jev still said "no clear emotion" 32% of the time.

What the data shows

Makima is listening meme: "Joan lived next to a dumpster." Jev: no clear emotion. — "Joan lived next to a dumpster." Jev: no clear emotion.

"No clear emotion" is Jev's most common answer. 32% of its answers, against 3% of the annotators'.
Even when people agree. On the 428 lines where two of three annotators named the same feeling, Jev picks "no clear emotion" 32% of the time.
Trust and surprise disappear. Annotators give trust 11% and surprise 13%; Jev 1% and 4%.
Joy it sees just as people do (24% each).

What it means, and what it doesn't

In stories, Jev waits to be told. Feelings that are implied rather than stated, especially trust and surprise, mostly don't register, and one answer in three is "no clear emotion". For a model that reads people's messages or stories, the unstated half of the emotional content goes missing.

It doesn't mean Jev is wrong every time. The annotators were asked to find feelings, and where it's known what writers actually felt, "nothing much" is sometimes the truth (see Caveats). This is a known tendency, measured here on stories rather than discovered.

Caveats

Annotators were asked to find a feeling. The annotators' task was to label each character's emotion, which may have pushed them away from "none". The gap is partly Jev being literal and partly annotators reading feelings in.
Writers themselves sometimes feel nothing. Where the storyteller's own feeling is known, "nothing much" is a real answer: in another experiment ("Does Jev read the writer, or the other readers?"), Jev's "no particular emotion" matches writers who felt nothing more often than other readers do.
A known tendency. Reading text literally, and not inferring what isn't stated, is on TypeSafe's own list of Jev's known weak spots. This experiment measures how large it is on stories; it isn't a new discovery.
Stories cut off mid-way. Each question shows the story only up to the line being annotated, as the annotators saw it. Early lines carry little emotional information, which invites "no clear emotion".

Jev on this experiment

Would a person find it interesting to read?: Yes69%
Does it describe you?: Yes74%
Would you have predicted it?: No51%
How fair is the comparison?: The comparison is reasonable
How much should a reader rely on it?: Moderately
Which caveat matters most?: Annotators were asked to find a feeling88%

Why ask this

Most of reading a story is filling in what isn't said. "Joan lived next to a dumpster. She never thought much about it until one particular day." Nobody says how Joan feels, but a reader starts guessing: dread, disgust, curiosity.

A reader that often answers "no clear emotion" is being literal where people infer. That's fine in a contract and a problem in a conversation, where most feelings are implied.

How this was done

The people and the data

StoryCommonsense (Rashkin and colleagues, 2018): five-sentence everyday stories, annotated line by line for each character's feelings by three paid crowd workers each on Amazon Mechanical Turk, using Plutchik's eight basic emotions (joy, trust, fear, surprise, sadness, disgust, anger, anticipation) plus "no clear emotion". The experiment uses 2,971 story lines, each shown only up to the line being labeled, as the annotators saw it.

What Jev was asked

The story up to the line in question, and the nine answers:

Which emotion best describes the feelings of Joan at the end of [story]?
joy or happiness · fear or worry · anger or annoyance · trust or acceptance · disgust · sadness · surprise · anticipation, looking forward to something · no clear emotion

For the dumpster story above, the annotators leaned sadness (61%) and disgust; Jev said no clear emotion (81%).

How it was measured

The share of all answers that went to each of the nine options, for Jev and for the annotators. Then the story lines where at least two of three annotators named the same real emotion: how often does Jev still say "no clear emotion"?

Where these questions live

2,971 questions across 1 topic of the map. Each opens on the map with every question in it.

Reading Feelings & Character

Story Character Feelings2,971

Every question

All 2,971 questions behind this result, the telling ones first: the examples the analysis points to, then the ones where Jev misses, biggest gap first.

Jev’s own answer

Showing 0 of 2,971