Atlas › Work tasks

Case study 48 of 198

Jev knows medicine, not the medical coder's rules

Asked which ICD-10-CM chapter a diagnosis belongs to, where does Jev go wrong: the medicine, or the coding conventions?

result2,000 questions

Jev files 87% of diagnoses in the right chapter of the ICD-10 medical code book, except for one kind: codes for how an injury happened (a fall, a crash, a bite). Those it gets right only 11% of the time, filing 86% of them under the injury itself. That's the medical answer, not the coder's.

external causes11%

health status factors74%

nervous system75%

symptoms signs findings75%

musculoskeletal78%

genitourinary80%

infectious parasitic84%

skin subcutaneous86%

blood immune86%

circulatory86%

perinatal87%

congenital88%

endocrine metabolic88%

respiratory89%

digestive91%

eye adnexa92%

ear mastoid94%

mental behavioral95%

injury poisoning96%

neoplasms96%

pregnancy childbirth98%

How to read this: Each bar is one chapter of the ICD-10-CM code book: how often Jev put that chapter's codes in it. The short bar at the top is "external causes", how an injury happened.

2,000 codes, about 95 per chapter.

In short

Jev files 87% of diagnosis codes in the right ICD-10 chapter, with cancers, injuries and pregnancy codes at 96% or better.
The big miss is a coding rule, not medicine: of 95 codes for how an injury happened, 82 went to the injury chapter instead.
Jev saw only the chapter titles and no coding guidelines; a prompt explaining the external-causes rule was not tested.

What the data shows

Stupid dumb drooling puzzle meme: A bus crash goes in the "external causes" chapter. Jev: the injury chapter (86% of the time). — A bus crash goes in the "external causes" chapter. Jev: the injury chapter (86% of the time).

The medicine is right. Across the other 20 chapters Jev is mostly correct: 98% for pregnancy and childbirth, 96% for cancers and for injuries, 95% for mental disorders.
The convention is missed. For external causes, 11% right. 82 of its 95 external-cause codes went to the injury chapter.
Next weakest: the grab-bags. "Factors influencing health status" (74%, sometimes filed under pregnancy) and "symptoms, signs and abnormal findings" (75%) are chapters defined by coding practice rather than by body system. The nervous system chapter is also at 75%.

What it means, and what it doesn't

Jev knows medicine well enough to sort diagnoses by body system, and it doesn't know the coder's rulebook. That's the typical gap for a general model in a regulated field: the domain knowledge is there, the conventions aren't. For coding work it needs the rules in the prompt (or in the option descriptions), not just the chapter names.

It doesn't mean Jev would get real codes wrong at this rate: coding a full record involves far more than the chapter, and a prompt that explains the external-causes rule might close much of this gap. That wasn't tested here.

Caveats

Descriptions only. Jev saw each code's official one-line description, not a patient record. Real coding starts from clinical notes, where the circumstances of an injury may be spelled out or missing.
The chapter names are what Jev had. Jev chose from the chapters' official titles, with no coding guidelines. A human coder learns that "how it happened" has its own chapter; from the titles alone that rule isn't obvious.
About 95 codes per chapter. Each chapter is represented by about 95 codes, spread across its categories. Per-chapter shares are rough to a few points.

Jev on this experiment

Would a person find it interesting to read?: Yes76%
Does it describe you?: Yes52%
Would you have predicted it?: Yes53%
How fair is the comparison?: The comparison is shaky
How much should a reader rely on it?: Moderately
Which caveat matters most?: The chapter names are what Jev had70%

Why ask this

Medical coding turns a diagnosis into a code from ICD-10, the classification hospitals and insurers use for billing and statistics. It's a common job for AI, and it follows rules that aren't purely medical. The clearest example: how an injury happened (a fall from a ladder, a car crash, a dog bite) is coded in its own chapter, external causes, separate from the injury itself (a fractured wrist goes in the injury chapter).

A model that reasons from the medicine will file "fell from a ladder" with the injury. A coder won't. That's a clean test of whether Jev knows the domain's conventions or just the domain.

How this was done

The people and the data

No people: the answer key is the code book itself. The project used the US edition (ICD-10-CM, fiscal year 2026, published by the Centers for Medicare and Medicaid Services), took about 95 codes from each of its 21 chapters, 2,000 in all, and asked Jev to place each by its description.

What Jev was asked

Each code was one pick-one question with the chapters' official titles:

Which ICD-10-CM chapter does the diagnosis belong to?
Diagnosis: "Driver of bus injured in collision with unspecified motor vehicles in traffic accident, initial encounter"
Options: the 21 chapter titles, from "Certain infectious and parasitic diseases" to "External causes of morbidity" and "Factors influencing health status and contact with health services"

That code belongs in external causes; Jev put it under injuries.

How it was measured

The share of codes Jev places in the right chapter, chapter by chapter, and the most common wrong chapter for each.

Where these questions live

2,000 questions across 1 topic of the map. Each opens on the map with every question in it.

Healthcare

Intake, Coding & Admin2,000

Every question

All 2,000 questions behind this result, the telling ones first: the examples the analysis points to, then the ones where Jev misses, biggest gap first.

Jev’s own answer

Showing 0 of 2,000