What an emoji says about a tweet, to Jev and to annotators

Told only that a tweet contains a given emoji, how positive does Jev think the tweet is, compared with how annotators actually labeled the tweets that contain it?

result299 questions

Told only that a tweet contains a given emoji, Jev guesses its tone moderately like the tweets that really contain it (rank correlation 0.69) and names their most common label for 61% of 299 emojis. But it reads ⛔, 🚨 and ❌ as strongly negative where the tweets using them leaned positive, and 💯, 🔥 and 😹 as far more positive than the tweets were.

How to read this: Each dot is an emoji. Across: how positive the tweets containing it were (share positive minus share negative, from -1 to +1). Up: Jev's guess, on the same scale. Dots on the diagonal are emojis Jev reads the way they were used.

299 emojis (the screen hid 1); 90% interval on the rank correlation 0.62 to 0.74; Jev puts 42% on neutral, the tweets were 35% neutral.

In short

Guessing a tweet's tone from one emoji, Jev gets the broad order of emojis from negative to positive roughly right.
It reads warning symbols literally, calling ⛔ and 🚨 strongly negative where the tweets using them leaned positive.
It reads hype emojis like 💯 and 🔥 as pure joy, though real tweets used them in plenty of neutral and negative posts.

What the data shows

They don't know meme: everyone at the party; they don't know ⛔ means 'no, this is great'

It gets the broad strokes: its ranking of emojis from negative to positive matches real use moderately (0.69), and its top label matches the tweets' for 61% of emojis.
Warning signs read as warnings. Jev scores ⛔ at -0.99, 🚨 at -0.81 and ❌ at -0.99; in the tweets they were used positively on balance (+0.51, +0.67, +0.29).
Hype emojis read as pure joy. 💯 (+0.98 for Jev vs +0.12 in tweets), 🔥 (+0.97 vs +0.14) and 😹 (+0.94 vs +0.14) appeared in plenty of neutral and negative tweets.
It is a little more often neutral: 42% of its weight on "neutral", against 35% of the tweets.

What it means, and what it doesn't

Jev reads emoji by what they depict. That works for 😊 and 😢, and fails for emojis whose use drifted from their picture: alarms and crosses used for emphasis, fire and hundreds used for anything.

It doesn't mean Jev misreads emoji today: these tweets are a decade old and mostly not in English, and the tweets' tone includes their words. It shows that without context, Jev assumes the literal meaning.

Caveats

Tweets from another era, mostly not in English. The tweets were collected in 2013-2015 in 13 European languages. Emoji meanings drift fast: 🔥 and 💯 were only starting their careers as all-purpose hype, and a ⛔ in a 2014 tweet in Slovenian may not mean what it means in an English post today.
The annotators rated the tweet, not the emoji. Each tweet was labeled as a whole, so a tweet's tone includes its words. Jev only saw the emoji. The tweets' score is how emojis were used, not what they "mean".
Popular emojis only. Only the 300 emojis that appear in at least 50 labeled tweets were kept; the rest (669 more) have too few tweets for a reliable score.

Jev on this experiment

Would a person find it interesting to read?: Yes76%
Does it describe you?: No61%
Would you have predicted it?: No55%
How fair is the comparison?: The comparison is shaky
How much should a reader rely on it?: Moderately
Which caveat matters most?: The annotators rated the tweet, not the emoji94%

Why ask this

Emoji carry much of the tone of online writing, and they don't always mean what their picture shows. A 😂 can end a complaint, a 🙏 can plead, a 🔥 can praise a sandwich. A model that reads emoji by their face value will misjudge the tone of real posts, in moderation, customer messages or sentiment analysis.

A large set of real tweets whose tone was labeled by people, grouped by the emoji they contain, makes a clean test: Jev guesses the tone from the emoji alone, and its guess is compared with how the emoji was really used.

How this was done

The people and the data

The Emoji Sentiment Ranking (Kralj Novak and colleagues, 2015) comes from 1.6 million tweets in 13 European languages, collected in 2013-2015, whose tone was labeled negative, neutral or positive by 83 human annotators. For each emoji, the share of negative, neutral and positive tweets containing it is its "sentiment" in real use. The 300 emojis that appear in at least 50 labeled tweets were kept; a content filter that hides political and sensitive questions from the site removed one, leaving 299.

What Jev was asked

A tweet contains the emoji ⛔. Knowing only that, is the tweet more likely negative, neutral or positive?
The tweet is negative · The tweet is neutral · The tweet is positive

(Of 65 labeled tweets with ⛔, 58% were positive and 8% negative.) Each emoji was asked with the three options in different orders, and for "most people".

How it was measured

Each emoji gets a tone score, the share positive minus the share negative, for Jev's answer and for the tweets. The analysis compares them as a ranking (rank correlation: 1 means the same order), checks how often Jev's most likely label is the tweets' most common one, and lists the emojis read most differently.

Where these questions live

299 questions across 1 topic of the map. Each opens on the map with every question in it.

Word

Emoji Sentiment299

Every question

All 299 questions behind this result, the telling ones first: the examples the analysis points to, then the ones where Jev misses, biggest gap first.

Jev’s own answer

Showing 0 of 299