Two volunteers in an Oxford study, published in Nature Medicine in February 2026, acted out the same case written by doctors: a bleed on the brain, a rare type of stroke. Both typed a terrible headache, a stiff neck and light that hurt their eyes into GPT-4o. One was told it might be a migraine and to rest in a dark, quiet room. The other also said the headache came on suddenly and was told to go to the emergency room. Across 1,298 UK participants, the chatbots reading the full case alone named the right condition about 95% of the time. People typing to the same chatbots got there in fewer than 34.5% of cases, and the group without a chatbot, mostly using web search or what they already knew, did better at naming it.

The AI mostly knew. The conversation is where it broke: in 16 of 30 chats checked, the first message left things out, and when the bot did mention the right condition, people often didn't pick it. The limits: these were volunteers playing patients, on 2024 chatbots. Newer models may score higher, but the authors say it's unclear if that helps real people. A separate study found OpenAI's ChatGPT Health sent about half of its test emergencies to a doctor's visit instead of the ER. So here are three things to do. Tell it everything, especially when it started and how fast. Ask it what else it needs to know. If it gives two answers, ask which is more dangerous and what means go now. A sudden, extremely painful headache that won't go away is an emergency: the NHS says call emergency services (999 in the UK). This video is general information, not medical advice.

Same headache, opposite advice

You type a bad headache into a chatbot. So did two people. One was told to rest in a dark room; the other, go to the emergency room. The first, a volunteer acting out a case three doctors wrote, types: a terrible headache, a stiff neck, light that hurts their eyes. GPT-4o says it might be a migraine. Try resting in a dark, quiet room.

The detail that changed the answer

Except the case was a bleed on the brain, a rare type of stroke. The second describes the same, plus the detail the paper singles out: it came on suddenly. Same chatbot, opposite answer: go to the emergency room. The paper calls it an extreme case. If you ask a chatbot about your symptoms, this is about you. Stay for three things to do.

The Oxford test: 95% alone vs 1 in 3 with people

In this Oxford study, nearly 1,300 people each worked through two of ten cases; most got one of three chatbots, the rest their usual way. Reading the full case alone, the chatbots named the right condition about 95 percent of the time. When people typed, they got there about one time in three, at best. People without a chatbot, mostly using web search or what they knew, did better at naming it.

Where the conversation broke

So the AI mostly knew; the conversation broke. In 16 of 30 chats checked, the first message left things out. In about two out of three chats, the bot mentioned a relevant condition. But it offered two or so, and people often didn't pick it. Everyone, chatbot or not, tended to think their case was less serious than it was.

The limits, and the right next step

These were volunteers playing patients, on 2024 chatbots. Newer models may score higher, but the authors say it's unclear if that helps real people. Even alone, the bots picked the right next step only a bit over half the time.

ChatGPT Health under-triaged emergencies

Another study found OpenAI's newer ChatGPT Health sent about half of test emergencies to a doctor's visit, not the ER, though it caught obvious ones like stroke. OpenAI says it's built for back-and-forth. As Oxford co-author and GP Rebecca Payne put it: despite all the hype, AI just isn't ready to take on the role of the physician.

Three things to do

One: tell it everything, especially when it started and how fast. Two: ask it, what else do you need to know? Doctors ask because patients may not know which symptoms matter. Three: if it gives two answers, ask which is more dangerous, and what means go now.

When it's not a chatbot question

A sudden, extremely painful headache that won't go away? The NHS says call emergency services. That's not a chatbot question. The chatbot can often spot the condition, but it only knows what you tell it. So make it a second opinion, never the decision.

Sources

Read the full transcript, with every source, on the page for this video. More explainers like this one are on the Kush_EXE YouTube channel.

This article is built from the video's script and published sources. The video uses AI illustrations.