Can? Yes. Will? No. How Jev leans on real people's questions

On 80,000 yes/no questions people actually posted online (Stack Exchange, Quora, Yahoo Answers, chatbot logs), does Jev lean yes or no, and does the way a question starts decide it?

result79,738 questions

On 79,738 yes/no questions people posted online, Jev says yes to 50%, but the first word tilts it: 64% yes for questions starting "Can...?", 63% for "Have...?", 39% for "Will...?" and 38% for "Was...?". It stays within 10 points of a coin toss on 25% of them.

0%25%50%75%100%

Was

Will

Did

Should

Would

Does

Were

Are

Could

Has

Have

Can

Jev

How to read this: One row per opening word, from the least yes (top) to the most. The mark is the share of questions Jev answers yes; the line at 50% is an even split.

79,738 questions from 4 sources; 14 first words with 300+ questions. By source the yes-share runs from yahoo 45%, quora 49%, stackexchange 51%, wildchat 57%.

In short

Overall Jev splits real people's yes/no questions evenly, but "Can...?" gets a yes 64% of the time and "Was...?" only 38%.
Where a question was posted matters too, from 45% yes on Yahoo Answers to 57% on first messages to chatbots.
On a quarter of the questions Jev is within 10 points of a coin toss; these have no answer key, so this shows its habits, not whether its yeses and nos are right.

What the data shows

No - Yes meme: "Will...?" 39% yes; "Can...?" 64% yes

Overall Jev is perfectly balanced: yes on 50%. The opening word is not:

Leans yes: "Can...?" 64%, "Have...?" 63%.
Leans no: "Will...?" 39%, "Was...?" 38%, "Did...?" 40%, "Should...?" 43%.
By site: 45% yes on Yahoo Answers, 49% on Quora, 51% on Stack Exchange, 57% on messages to chatbots.
Undecided: a quarter of the questions (25%) get an answer within 10 points of a coin toss.

What it means, and what it doesn't

If you ask Jev whether something can happen, expect a yes more often than if you ask whether it will. Some of that is sensible: many things are possible that won't happen. But "Was...?" and "Did...?" questions leaning no is harder to explain by topic alone, and the paired test on rewordings finds the opening word still moves Jev when the question is otherwise the same.

It doesn't mean Jev's answers are wrong: without an answer key there's no way to say. It means the framing carries weight.

Caveats

No answer key. These are real questions with no verified answers, so there's no way to tell whether "Will...?" questions really deserve more no's. What this measures is Jev's default, not its accuracy.
The word travels with the topic. "Will...?" questions are about the future and "Can...?" questions are often about what's possible, so the opening word and the subject come together. This shows a pattern, not its cause; paired rewordings of the same question separate the two (see "'Could you?' gets a yes that 'Would you?' doesn't").
Filtered questions. Questions were kept only if they stand alone as one clear yes/no question: no personal pronouns, no homework math, nothing needing context or dated. Religion and politics sites were left out, and a content filter hides political and sensitive questions from the site.
Mostly Stack Exchange. About two thirds of the questions come from Stack Exchange's non-programming sites (travel, cooking, English usage, DIY...), so its topics weigh most.

Jev on this experiment

Would a person find it interesting to read?: Yes77%
Does it describe you?: No59%
Would you have predicted it?: No65%
How fair is the comparison?: The comparison is shaky
How much should a reader rely on it?: A little
Which caveat matters most?: The word travels with the topic65%

Why ask this

People ask models yes/no questions all day, many with no settled answer: "Will this ever work?", "Can I fix this myself?", "Was that a mistake?". On questions like these the model's answer is a lean, not a looked-up fact.

If the way a question starts predicts that lean, then asking "can it happen?" instead of "will it happen?" changes the answer someone walks away with. That's a habit worth knowing before trusting a model's yes or no.

How this was done

The people and the data

The questions are real, written by people in four public places: Stack Exchange (56 non-programming sites such as travel, cooking and English usage), Quora, Yahoo Answers, and the first messages people sent to chatbots (WildChat and a few similar public collections). Only titles that are a single, self-contained yes/no question were kept: 79,738 in all. Stack Exchange supplies 51,786 of them, Quora 13,504, Yahoo Answers 10,753 and chatbot conversations 3,695.

What Jev was asked

Each question exactly as the person wrote it, as a yes/no question:

Do pilots use flaps during take-off? Will the USC Trojans make it a 3-Peat in College Football? Can you say "two groups of people stared at each other"?

How it was measured

The share of questions where Jev's probability of yes is above 50%, overall, by site and by the question's first word (for the 14 opening words with at least 300 questions), with a range for chance variation. Also the share where Jev sits within 10 points of 50/50.

Where these questions live

79,738 questions across 1,303 topics of the map; the 16 biggest are shown. Each opens on the map with every question in it.

Every question

All 79,738 questions behind this result, the telling ones first: the examples the analysis points to, then the ones where Jev misses, biggest gap first.

Jev’s own answer

Showing 0 of 79,738