To Jev, a stirring word is an unpleasant one

When Jev rates how calming or stirring a word feels, is it rating excitement, as people do, or just how pleasant the word is?

result1,714 questions

For Jev, a stirring word is mostly an unpleasant one. Its ratings of how calming or stirring a word feels follow people's pleasantness ratings in reverse (rank correlation -0.43) about as closely as they follow people's own calm-or-stirring ratings (0.37). So it calls "cuddle" and "loving" calming, where people find them stirring, and "puke" and "misery" intensely stirring, where people rate them fairly calm (about 3 of 9).

How to read this: Each dot is a word. Across: how stirring people rated it. Up: how stirring Jev rated it. If Jev read excitement the way people do, the dots would climb from bottom left to top right.

468 words rated on both scales; 1,246 arousal ratings for the gap lists.

In short

When Jev rates how stirring a word feels, it largely answers how unpleasant the word is (-0.43), about as closely as it follows people's own stirring ratings (0.37).
Pleasant, lively words like "cuddle" land at calm for Jev (people: 6.6 of 9), while "misery" lands near the top (people: about 3.3).
The project's own answer wording illustrated "stirring" mostly with unpleasant examples, which may explain part of the gap.

What the data shows

Spongebob Yelling meme: The word "misery" to Jev: JOLTS YOU AWAKE — The word "misery" to Jev: JOLTS YOU AWAKE

Jev's arousal ratings track people's pleasantness in reverse (-0.43) about as strongly as they track people's own arousal ratings (0.37). For people, the dials are moderately linked the other way: pleasant words lean stirring (+0.37). For Jev, the two collapse into one: its own arousal and its own pleasantness ratings run opposite each other at -0.50.

The words it gets most wrong show the pattern plainly:

Pleasant words it calls calming that people find stirring: cuddle, loving, snuggle, caress, warmth, beach. People put "cuddle" at 6.6 of 9; Jev put it at the very bottom, "sleepy or soothing".
Unpleasant words it calls intensely stirring that people rate calm to middling: puke, misery, awful, amputate, anguish, deluge. People rated "misery" about 3.3 of 9 for arousal: sad, but not jolting. Jev put it near the top.

What it means, and what it doesn't

When Jev says something is "intense" or "exciting", it's partly telling you it's unpleasant. That matters for any task that asks a model to rate energy separately from mood: music, ads, customer messages, even alarm levels.

It doesn't mean Jev can't tell pleasant from unpleasant. Its pleasantness ratings match people's closely (see "Jev knows how pleasant a word is, not how exciting"). The confusion is specific to the second dial. Part of it may also come from the project's own answer wording (see Caveats); rerunning the arousal questions with neutral examples is the obvious next test.

Caveats

The wording may be part of the effect. The five answer levels were written for this project, and their examples lean one way: "calming" is illustrated with a quiet evening, "intensely stirring" with danger, a thrill or a scream. Two of those three are unpleasant. That could nudge any reader, model or person, to hear "stirring" as "bad". The original study used its own scale instructions, not these, so part of the gap may be the project's phrasing rather than Jev.
Who the people are. The Glasgow Norms were rated by native English speakers from the University of Glasgow community, recruited through the psychology department, about 33 per word. Words like "beach" or "cuddle" may stir different feelings elsewhere.
Two scales squeezed into one. People rated on 1 to 9; Jev picked one of five described levels. Jev's levels are mapped onto 1 to 9 in equal steps, which is approximate. The rank correlations don't depend on that mapping; the gap lists do.
Which words. Only words asked both ways (pleasant and stirring) count, and words a content filter hides from the site as sexual or violent are missing, which removes some of the most arousing words people rated.

Jev on this experiment

Would a person find it interesting to read?: Yes74%
Does it describe you?: No61%
Would you have predicted it?: No73%
How fair is the comparison?: The comparison is shaky
How much should a reader rely on it?: A little
Which caveat matters most?: The wording may be part of the effect99%

Why ask this

Psychologists describe the feeling of a word on two separate dials. One is pleasantness: "sunshine" is pleasant, "vomit" is not. The other is arousal: how calm or stirred up the word makes you. The two are separate. "Cuddle" is pleasant and stirring; "boredom" is unpleasant and calm; "funeral" is unpleasant and stirring; "nap" is pleasant and calm.

A model that talks about feelings all day should keep those dials apart. If it quietly fuses them, treating "stirring" as a polite word for "unpleasant", then every time it's asked how exciting, intense or alarming something is, it's really answering a different question: how bad is it?

How this was done

The people and the data

The comparison is the Glasgow Norms (Scott, Keitel, Becirspahic, Yao and Sereno, 2019), one of the standard word-rating datasets in psychology: 5,553 English words, each rated on nine dimensions including pleasantness and arousal, on 1 to 9 scales. The raters were native English speakers from the University of Glasgow community, 829 people in all, about 33 per word. The study uses the average rating per word.

Of the words Jev was asked about, 468 were asked on both dials, which is what this comparison needs: the same word, rated for pleasantness and for arousal, by Jev and by people.

What Jev was asked

Each word was a separate question, with five described answers instead of numbers:

How calming or stirring does the word "cuddle" feel to you?
Calming: it feels sleepy or soothing, like a quiet evening · Mostly calm: it stirs little, like an everyday object on a shelf · Neither: it is no more calming than stirring · Somewhat stirring: it raises interest or alertness, like good news or a warning sign · Intensely stirring: it jolts you awake, like danger, a thrill or a scream

The pleasantness question was built the same way ("How pleasant or unpleasant does the word feel to you?"). Jev also answered both for "most people", and with the answer order reversed, to check that the order didn't drive it.

How it was measured

Each word has four numbers: how stirring Jev finds it, how stirring people find it, and how pleasant each finds it. The words are ranked on each and the rankings compared (a rank correlation: 1 means the same order, 0 means no relation, -1 means reversed). The telling comparison is Jev's arousal against people's pleasantness: if Jev's "stirring" is really "unpleasant", that pair will be strongly negative.

Where these questions live

1,714 questions across 142 topics of the map; the 16 biggest are shown. Each opens on the map with every question in it.

Every question

All 1,714 questions behind this result, the telling ones first: the examples the analysis points to, then the ones where Jev misses, biggest gap first.

Jev’s own answer

Showing 0 of 1,714