People are easy; brands and products in tweets are not

Given a name in a sentence, can Jev say what kind of thing it names, in edited news text and in tweets?

result4,447 questions

Jev knows a person's name when it sees one: 99% right in news and 97% in tweets. But in tweets it gets companies right only 69% of the time, products 72% and groups 76%. A company is most often taken for a product, and a product for a creative work.

other (news)81%

organization (news)90%

location (news)92%

person (news)99%

corporation (tweets)69%

product (tweets)72%

group (tweets)76%

creative work (tweets)88%

location (tweets)88%

person (tweets)97%

How to read this: Each bar is one kind of name, in edited news or in tweets: how often Jev typed it right. People are near the top everywhere; the brand-like names in tweets are the short bars.

4,447 names (2,488 news, 1,959 tweets).

In short

Jev types people's names almost perfectly, 99% right in news stories and 97% in tweets.
In tweets, brand-like names blur: corporations 69% right (usually called products), products 72% (usually called creative works), groups 76%.
The tweet set was built from rare and emerging names on purpose, so it is harder than everyday social media.

What the data shows

work tasks

Jev holding every brand name in a tweet (corporations: 69%)

How funny is this meme? Jev: 3/5, funny11%229%366%44%50%

People: easy. 99% in news, 97% in tweets.
News: solid. Locations 92%, organizations 90%, the catch-all "other" 81%.
Tweets: brands blur. Corporations 69% (most often called a product), products 72% (most often a creative work), groups 76% (most often a person).
Places in tweets hold up. 88%, about as well as creative works.

What it means, and what it doesn't

On social media text, Jev is reliable for people and places and shaky for anything commercial. Tools that track brand mentions or route product complaints from social posts will mix up companies, their products and their media; a company list or product catalog in the prompt would likely help.

It doesn't mean Jev is weak at names in general: in edited news it's near the top on every type but the catch-all.

Caveats

Tweets chosen to be hard. The tweet dataset (WNUT-17) was built on purpose from rare and newly emerging names, so it's harder than everyday social media.
Brand names are genuinely ambiguous. "Can't believe the Super City is nearly open!" labels Super City a corporation; Jev said a location (77%). Without knowing it's a store's name, many readers would say the same. Companies, their products and their apps often share a name.
Different menus. News names were sorted into four types, tweet names into six, so the two corpora aren't directly comparable beyond people and places.

Jev on this experiment

Would a person find it interesting to read?: Yes70%
Does it describe you?: No57%
Would you have predicted it?: Yes65%
How fair is the comparison?: The comparison is shaky
How much should a reader rely on it?: Moderately
Which caveat matters most?: Brand names are genuinely ambiguous72%

Why ask this

Deciding what a name refers to (a person, a place, a company, a product) is a basic building block for search, moderation, analytics and customer support. In edited news, names follow conventions: capitalized, introduced, with context. On social media, companies, their products, bands and apps share names, get abbreviated, and appear without introduction.

A brand-monitoring tool that can't tell a company from its product, or a band from its album, files every complaint under the wrong heading. So the question is where Jev's typing holds up and where it breaks.

How this was done

The people and the data

Two classic public datasets, 4,447 names:

CoNLL-2003: English news stories, with each name tagged as a person, organization, location or other.
WNUT-17: tweets and other social posts, built around rare and emerging names, tagged as a person, location, corporation, product, creative work or group.

The tags come from each dataset's annotators. The news half is balanced, 621 to 623 names of each type (2,488 in all). The tweet half (1,959 names) is less even, from 225 corporations and 225 products to 666 people.

What Jev was asked

Each name was one pick-one question with its sentence:

What type of entity is "Super City" in this sentence?
Sentence: "RT @tommcfly: Working on some final Super Site stuff all day. Can't believe the Super City is nearly open!"
Options: group (a band, sports team, political party or other group of people that is not a company) · person · product · location · corporation · creative work

How it was measured

The share of names Jev types right, per type and corpus, and the most common wrong type for the weakest ones.

Where these questions live

4,447 questions across 2 topics of the map. Each opens on the map with every question in it.

Structured Extraction

Named Entity Type2,488 Emerging Entity Type1,959

Every question

All 4,447 questions behind this result, the telling ones first: the examples the analysis points to, then the ones where Jev misses, biggest gap first.

Jev’s own answer

Showing 0 of 4,447