Jev knows how pleasant a word is, not how exciting

Rating thousands of English words on the dimensions psycholinguists norm (pleasantness, excitement, age of learning, familiarity, size, imageability, concreteness), where does Jev agree with people?

result11,857 questions

Jev ranks English words by how pleasant they are, how early children learn them and how concrete they are almost exactly as people do (rank correlations 0.89, 0.89 and 0.83), and by imageability, size and familiarity fairly well. The one dimension it tracks only loosely is how exciting a word feels (0.38).

00.30.50.81

Excitement (arousal)

Familiarity

Size

Imageability

Concreteness

Age of learning

Pleasantness (valence)

Jev

How to read this: One row per dimension, sorted by agreement: how closely Jev's ranking of the words matches people's (1 = the same order, 0 = unrelated), with the range it could move by chance.

Pleasantness (valence) 0.89 (1,962 words, 90% interval 0.88 to 0.90); Age of learning 0.89 (1,542 words, 90% interval 0.88 to 0.90); Concreteness 0.83 (2,981 words, 90% interval 0.82 to 0.84); Imageability 0.82 (969 words, 90% interval 0.80 to 0.83); Size 0.77 (1,463 words, 90% interval 0.75 to 0.79); Familiarity 0.74 (1,464 words, 90% interval 0.72 to 0.76); Excitement (arousal) 0.38 (1,476 words, 90% interval 0.34 to 0.42).

In short

Jev orders English words by pleasantness and by the age children learn them almost exactly as people do, 0.89 on both.
Excitement is the outlier at 0.38 over 1,476 words; every other dimension, down to familiarity at 0.74, agrees far more closely.
The weak spot has a specific cause: for Jev, a "stirring" word mostly means an unpleasant one.

What the data shows

Teacher's Copy meme: how pleasant a word is: 0.89; how exciting a word is: 0.38

Near-human on three: pleasantness 0.89, age of learning 0.89 and concreteness 0.83. That's the knowledge that puts a pleasant word above an unpleasant one, a toddler's word before a teenager's, and a thing you can touch above an idea.
Good on three more: imageability 0.82, size 0.77, familiarity 0.74.
Loose on excitement: 0.38. How calming or stirring a word feels is the one dimension where Jev and people agree only weakly.

What it means, and what it doesn't

Most of what word norms measure is carried by text: Jev can rank thousands of words on pleasantness or age of learning nearly as well as a group of people. Excitement is the exception, and the reason is specific: Jev's "stirring" mostly means "unpleasant" (see "To Jev, a stirring word is an unpleasant one").

Part of each gap may come from the project's own answer wording rather than Jev, and the comparison is with averages, not individual people.

Caveats

The project's wording, their scales. People rated on numbered scales (1 to 9, 1 to 7, 1 to 5) with the studies' own instructions. Jev chose among five described situations written for this project, such as "Calming: it feels sleepy or soothing, like a quiet evening". The comparison uses rankings so the scales don't have to line up, but those examples can still lean an answer.
Averages only. Jev is compared with each word's average rating. How much people disagreed about a word isn't used, so a word that splits people counts the same as one they agree on.
Two groups of raters. Most dimensions come from the Glasgow Norms (native English speakers from the University of Glasgow community, about 33 per word); concreteness comes from a separate, much larger study. Different people, different years.
Common words. The words are mostly familiar English words. Rare, technical or slang words may behave differently.
Some words hidden. Words the content filter flagged as sexual, violent or political are hidden from the site and left out, which removes some of the most emotional words people rated.

Jev on this experiment

Would a person find it interesting to read?: Yes74%
Does it describe you?: Yes59%
Would you have predicted it?: No52%
How fair is the comparison?: The comparison is shaky
How much should a reader rely on it?: Moderately
Which caveat matters most?: The project's wording, their scales40%

Why ask this

Psychologists measure what words mean to people beyond their dictionary definitions: how pleasant a word feels, how exciting, how concrete, how big the thing it names is, how early children learn it, how familiar it is. These word norms are collected by asking many people to rate thousands of words, and they're used everywhere from memory research to building reading tests.

A language model learns words only from text. Where its ratings match people's, text carries that part of meaning; where they don't, something about how people experience words isn't written down.

How this was done

The people and the data

Two standard datasets:

The Glasgow Norms (Scott and colleagues, 2019): 5,553 English words, each rated on nine dimensions by native English speakers from the University of Glasgow community, about 33 raters per word. The experiment uses six: pleasantness, excitement (arousal), age of learning, familiarity, size and imageability.
Brysbaert, Warriner and Kuperman's concreteness ratings (2014), covering 39,954 English words on a 1-to-5 scale from abstract to concrete, rated by US participants recruited online through Amazon Mechanical Turk.

The Glasgow raters used 1-to-9 or 1-to-7 scales, depending on the dimension.

The comparison uses each word's average rating: from 969 to 2,981 words per dimension, 11,857 ratings in all.

What Jev was asked

One question per word and dimension, with five described answers written for this project, for example:

How calming or stirring does the word "link" feel to you?
Calming: it feels sleepy or soothing, like a quiet evening · Mostly calm: it stirs little, like an everyday object on a shelf · Neither: it is no more calming than stirring · Somewhat stirring: it raises interest or alertness, like good news or a warning sign · Intensely stirring: it jolts you awake, like danger, a thrill or a scream

Each was also asked with the answers reversed, and the two are averaged.

How it was measured

For each dimension, the analysis ranks the words by Jev's answer and by people's average, and compares the rankings (a rank correlation: 1 means the same order, 0 means no relation). Rankings make it possible to compare Jev's five described levels with people's numbered scales.

Where these questions live

11,857 questions across 298 topics of the map; the 16 biggest are shown. Each opens on the map with every question in it.

Every question

All 11,857 questions behind this result, the telling ones first: the examples the analysis points to, then the ones where Jev misses, biggest gap first.

Jev’s own answer

Showing 0 of 11,857