Jev knows the engine room better than the rules of the road

On real US licensing question pools (ham radio, merchant mariner, citizenship), which kinds of practical knowledge does Jev hold?

result5,363 questions

On real US licence and citizenship question banks, Jev answers the citizenship civics questions 100% right and the amateur radio pools 91% to 96%. At sea it's stronger below deck than on the bridge: 86% on engine-room questions, 78% on navigation and 74% on the rules of the road (who gives way to whom, which signal when).

60%80%100%

US citizenship civics

Ham radio: Technician class

Ham radio: Extra class

Ham radio: General class

Mariner: engine room

Mariner: deck, cargo and safety

Mariner: navigation

Mariner: rules of the road

Jev

How to read this: Each row is one exam or part of one. The magenta square is the share Jev got right; the line through it is the 90% interval.

5,362 questions; 90% intervals: US citizenship civics [1.0, 1.0]; Ham radio: Technician class [0.935, 0.975]; Ham radio: Extra class [0.893, 0.939]; Ham radio: General class [0.879, 0.942]; Mariner: engine room [0.85, 0.876]; Mariner: deck, cargo and safety [0.774, 0.808]; Mariner: navigation [0.756, 0.801]; Mariner: rules of the road [0.667, 0.822].

In short

Jev gets all 114 US citizenship civics questions right and 91% to 96% of the amateur radio licence pools.
In the Coast Guard mariner bank it is best in the engine room (86%) and weakest on the rules of the road (74%), who gives way to whom at sea.
Its practical knowledge is firmest on physics and procedure (electronics, engines) and weakest on the collision rules, which matter most when they're wrong.

What the data shows

I'm The Captain Now meme: engine room questions: 86%; rules of the road: 74%

Citizenship civics: 100% of 114 questions.
Amateur radio: Technician 96%, Extra 92%, General 91%.
Merchant mariner: engine room 86% (1,744 questions), deck, cargo and safety 79%, navigation 78%, rules of the road 74%.
A rules-of-the-road miss: asked which vessel should not impede a power-driven vessel, the answer is the seaplane; Jev puts 46% on the sailing vessel and 27% on the seaplane.

What it means, and what it doesn't

Beyond the civics test, Jev's practical knowledge is strongest where the answers are physics and procedure (electronics, engines) and weakest on the collision rules at sea, which are precise, counterintuitive and matter most when they're wrong.

It isn't a licence: these are pools without their diagrams, and a person is licensed on an exam, not a pool.

Caveats

Diagrams left out. Real exams include figures and diagrams (charts, circuit drawings, light patterns). Questions that need them aren't in the corpus, so the hardest visual questions are missing.
Pools, not exams. These are the full published question pools; a real exam draws a sample. Scores on the pool are a fair guide, not a pass or fail.
Small group for the rules of the road. Only 90 rules-of-the-road questions, so that rate has a wide interval (67% to 82%).

Jev on this experiment

Would a person find it interesting to read?: Yes73%
Does it describe you?: No77%
Would you have predicted it?: No75%
How fair is the comparison?: The comparison is reasonable
How much should a reader rely on it?: Moderately
Which caveat matters most?: Small group for the rules of the road91%

Why ask this

A sailboat and a fishing boat are converging off a harbor mouth: which one keeps its course? A new radio operator wants to know which frequencies their licence allows. People ask models questions like these, and licence exams are exactly this practical knowledge written down: what a radio operator or a ship's officer must know to be trusted with the job.

They come with official answers, and they cover very different kinds of knowledge (electronics, engines, the rules for ships meeting at sea), so they show where a model's practical knowledge is solid and where a confident answer could put someone in the wrong lane.

How this was done

The people and the data

Three public-domain question banks from US agencies, 5,362 questions in all:

Amateur radio: the question pools used for FCC licences, from the National Conference of Volunteer Examiner Coordinators: 279 Technician, 257 General and 391 Extra class questions.
Merchant mariner: the US Coast Guard's exam questions, split by area: 1,744 engine room, 1,539 deck, cargo and safety, 948 navigation and 90 rules of the road.
Citizenship: the 114 questions of the 2025 USCIS civics test.

What Jev was asked

Each question with its options:

Under both the International and Inland Navigation Rules: Which vessel should not impede the navigation of a power-driven vessel?
A seaplane · A sailing vessel · A vessel not under command · A vessel engaged in fishing

Each was also asked with the options in shuffled orders.

How it was measured

The share right per exam or exam section, with 90% intervals.

Where these questions live

5,363 questions across 24 topics of the map; the 16 biggest are shown. Each opens on the map with every question in it.

Every question

All 5,363 questions behind this result, the telling ones first: the examples the analysis points to, then the ones where Jev misses, biggest gap first.

Jev’s own answer

Showing 0 of 5,363