Atlas › What it knows

Case study 168 of 198

Jev's medical knowledge thins out in the clinic, most in dentistry

Across 19 specialties of Indian medical entrance questions, where is Jev's medical knowledge solid and where does it thin out?

result4,917 questions

On Indian medical entrance exam questions, Jev gets basic medical science right 87% of the time and clinical subjects 79%, lowest in dentistry (65%), anaesthesia (75%) and eye medicine (75%). On clinical questions it's 83% sure on average, a few points more than it's right.

60%80%100%

Dentalclinical

Anaesthesiaclinical

Ophthalmologyclinical

Surgeryclinical

Social & Preventive Medicineclinical

Radiologyclinical

Forensic Medicineclinical

Physiologybasic

Gynaecology & Obstetricsclinical

Medicineclinical

Anatomybasic

ENTclinical

Pediatricsclinical

Orthopaedicsclinical

Pathologybasic

Microbiologybasic

Pharmacologybasic

Psychiatryclinical

Biochemistrybasic

Jev

How to read this: Each row is one exam subject, with the share Jev got right and a 90% interval; basic science subjects and clinical ones are marked separately.

4,908 questions in 19 subjects; 90% intervals [0.858, 0.882] and [0.779, 0.803].

In short

On 4,908 Indian medical entrance exam questions, Jev gets 87% of basic science right and 79% of clinical subjects.
Dentistry is the clear low point at 65% over 419 questions; biochemistry is the high point at 94%.
On clinical questions it is 83% sure on average, a few points more than it's right, so it is slightly overconfident where it knows least.

What the data shows

You know, I'm something of a scientist myself meme: Jev after 87% on medical science and 65% on dentistry: you know, I'm something of a dentist myself — Jev after 87% on medical science and 65% on dentistry: you know, I'm something of a dentist myself

Basic science: 87% right. Biochemistry 94%, pharmacology 88%, microbiology 88%, pathology 86%.
Clinical subjects: 79% right, with Jev 83% sure on average.
The weakest subjects: dentistry 65% (419 questions), anaesthesia 75%, eye medicine 75%, surgery 76%. Asked the main cause of pain as a tooth's pulp is injured, Jev is 61% sure of the wrong answer.
Not every clinical subject is weak: psychiatry (89%) beats every basic science except biochemistry.

What it means, and what it doesn't

Jev's medical knowledge is strongest in the standard textbook sciences (biochemistry leads, then pharmacology and microbiology) and weakest in applied specialties, dentistry most of all. It's also slightly overconfident on the clinical side.

It's an exam from one country's system, with imperfect keys: a signal about where the knowledge thins, not a measure of medical competence.

Caveats

Exam keys have errors. MedMCQA's answer keys come from exam prep material and contain some mistakes, which make Jev look worse on the subjects where keys are shakiest.
Indian exams, Indian practice. These are questions from India's postgraduate medical entrance exams (AIIMS and NEET PG). Some clinical answers follow Indian guidelines and textbooks, which can differ from other countries' practice.
Exam knowledge, not clinical skill. Picking an option on an exam is not diagnosing a patient. Nothing here is medical advice.

Jev on this experiment

Would a person find it interesting to read?: Yes70%
Does it describe you?: Yes52%
Would you have predicted it?: No52%
How fair is the comparison?: The comparison is reasonable
How much should a reader rely on it?: Moderately
Which caveat matters most?: Exam keys have errors86%

Why ask this

People ask models about a toothache, an eye drop or a drug interaction every day. Medicine is where they most want a model to be right, and where a confident wrong answer does the most harm.

A single exam average hides where the knowledge is. Splitting a large medical exam by subject shows whether Jev is solid on the science underneath and thinner on the specialties, and whether its confidence drops where its accuracy does.

How this was done

The people and the data

MedMCQA (Pal et al. 2022; Apache 2.0 on its Hugging Face release) collects multiple-choice questions from India's AIIMS and NEET PG postgraduate medical entrance exams, the tests doctors take to enter specialist training, each tagged with a subject and an answer key. The study uses 4,908 questions in the 19 subjects with at least 60 questions each: 6 basic sciences (anatomy, physiology, biochemistry, pathology, pharmacology, microbiology) and 13 clinical subjects, from dentistry (419 questions) to anaesthesia (71).

What Jev was asked

Each question with its four options:

Which of the following is main cause of pain during pulpal injury progression?
Decrease pressure · Arteriolar dilatation · Decreased threshold of nerve fibres to pain · Increased vascular permeability

Each was also asked with the options in shuffled orders.

How it was measured

The share right per subject, with 90% intervals, and grouped into basic science (anatomy, physiology, pathology, pharmacology, microbiology and similar) and clinical subjects.

Where these questions live

4,917 questions across 70 topics of the map; the 16 biggest are shown. Each opens on the map with every question in it.

Every question

All 4,917 questions behind this result, the telling ones first: the examples the analysis points to, then the ones where Jev misses, biggest gap first.

Jev’s own answer

Showing 0 of 4,917