AI chatbots pass medical exams, yet people using them don’t diagnose better

Key takeaways

• In a KFF poll taken in early 2026, 32% of U.S. adults said they had used an AI chatbot for health information or advice in the past year, about the same share that used social media for it.
• In a randomized Oxford-led study of 1,298 adults, chatbots alone named a relevant condition in 94.9% of cases, but people using them did so in fewer than 34.5%, no better than people using their usual sources.
• Chatbots can build confident answers on made-up details, and a 2023 study found that 18% to 55% of the references they listed did not exist.
• In August 2025, Illinois enacted a law barring AI from providing therapy, and in September 2025 the FTC opened a study of how companion chatbots affect children and teens.
• Use AI to learn vocabulary and prepare questions, then check what it says against a trusted source like MedlinePlus and a clinician who can examine you.

Two people in a British study typed in nearly the same story. Each described symptoms of a subarachnoid hemorrhage, a type of bleeding around the brain that needs emergency care.

One was told to lie down in a dark room. The other was told, correctly, to get emergency help².

The study, published in Nature Medicine in February 2026, was led by researchers at the University of Oxford¹. It tested three popular AI models: GPT-4o, Llama 3 and Command R+². On their own, the models identified a relevant condition in 94.9% of cases².

Then real people used them. Adults working through the same 10 doctor-written scenarios with a chatbot’s help named a relevant condition in fewer than 34.5% of cases. They picked the right next step, such as staying home or going to the ER, in fewer than 44.2%². They did no better than a comparison group who used whatever they normally would, such as a search engine².

That gap, between what a chatbot knows and what you walk away with, is what matters most for anyone typing symptoms into one. This guide covers the research and how to get the useful part without the risky part.

One in three adults now asks AI about their health

KFF, a nonpartisan health research group, surveyed 1,343 U.S. adults in late February and early March 2026³. Nearly a third, 32%, had used AI chatbots or tools for physical or mental health information in the past year³.

That’s on par with social media. Doctors and other providers are still the top source, used by 80%, followed by search engines at 68%³.

People mostly asked about symptoms. About 1 in 5 adults used AI to explain tests, lab results or diagnoses, and a similar share used it to compare treatment options³. About 16% used it to decide whether to see a doctor at all³.

Among users, 65% said quick answers were a major reason, and 36% said they felt more comfortable looking things up privately³. More than a quarter, 28%, had gotten test results before they could talk them over with a provider³.

Cost and access mattered too, especially for the young. Among users ages 18 to 29, 38% said a major reason was having no regular provider or no available appointment³.

Most users followed up. Among people who used AI for physical health questions, 58% later saw a doctor or other provider³. But 21% of adults ages 18 to 29 used AI for physical health and did not follow up, about twice the rate for older adults³.

Trust splits sharply. Among people who used AI for physical health, 69% trusted it at least a fair amount for reliable information. Among non-users, only 18% did³.

Doubts are not new. In KFF’s 2024 poll, 63% of adults were not confident that chatbot health information was accurate, and 56% were not confident they could tell true from false in chatbot answers⁴.

The answers can read better than a busy doctor’s

Part of the appeal is how chatbot answers sound. A 2023 study in JAMA Internal Medicine compared them with real doctors’ replies⁵.

The research team pulled 195 questions from Reddit’s r/AskDocs, a forum where verified physicians answer the public⁵. They gave the same questions to ChatGPT in December 2022. Licensed health professionals then rated both sets of answers, three graders per question⁵.

The graders preferred the chatbot’s answer 78.6% of the time⁵. They rated 78.5% of the chatbot answers good or very good in quality, compared with 22.1% of the doctors’ answers⁵. On empathy, 45.1% of the chatbot replies were rated empathetic or very empathetic, versus 4.6% of the doctors’⁵.

There’s a catch in the setup. The doctors’ replies averaged 52 words; the chatbot’s averaged 211⁵. Physicians typing quick answers to strangers online are not the same as your own doctor in an exam room. The study measured how answers read, not whether anyone got healthier, and the authors called for clinical trials to find out⁵.

So a chatbot can explain things clearly and kindly. That’s real value. It isn’t the same as being right about you.

Why knowing medicine isn’t the same as helping you

The Oxford team looked closely at 30 of the conversations to see where things broke down². The problems ran in both directions.

People left things out. In 16 of those 30 chats, the person’s first message gave only part of the picture². A doctor would ask follow-up questions to fill the gaps. Users, the Oxford team said, often didn’t know which details the chatbot needed¹.

Good answers got lost. The chatbots mentioned a relevant condition in roughly two-thirds to three-quarters of conversations². Yet people often didn’t carry that answer into their final decision. Only 34% of all the conditions the chatbots suggested were correct, so the right one was buried among wrong ones².

Small wording changes flipped the advice. The brain-bleed example above is one case². The Oxford researchers said slight differences in how a question was phrased could produce very different answers¹.

Some errors were basic. In one conversation, a model suggested a partial U.S. phone number and also told the British user to call “Triple Zero,” Australia’s emergency number².

People in the comparison group were more likely to spot the right condition. Their odds of naming a relevant one were 1.76 times higher than the chatbot users’, and they were more likely to catch the “red flag” conditions that need fast care².

The authors’ main warning was about testing. On medical-exam-style questions, the models typically cleared the rough passing mark of 60%, but those scores didn’t predict how they performed with real people². The researchers urged testing with real users before chatbots are used for patient care, much as new medicines are tested¹.

These were 2024-era models tested in the United Kingdom, and newer versions may do better.

Confident errors: invented details and fake sources

Chatbots tend to accept the premise of your question. Researchers at Mount Sinai in New York tested this in a study published in Communications Medicine in August 2025⁶.

They wrote fictional patient cases, each containing one made-up detail: a disease, symptom or lab test that doesn’t exist⁶. Leading chatbots often ran with the invented term, explaining it confidently as if it were real⁶.

The fix was surprisingly simple. Adding a single sentence warning the chatbot that the information might be inaccurate cut those errors nearly in half, according to senior author Eyal Klang⁶. That’s a tip you can use, and it’s built into the routine below.

Sources are another weak spot. In a 2023 study in Scientific Reports, two researchers asked ChatGPT to write short literature reviews on 42 topics and checked all 636 references it listed⁷.

With the older GPT-3.5 model, 55% of the references were fabricated. With GPT-4, 18% were. Even the real references often had substantive errors: 43% of GPT-3.5’s and 24% of GPT-4’s⁷. A citation from a chatbot is a lead to check, not proof.

Mental health is where the stakes are highest

In the KFF poll, 16% of adults had used AI for mental health information or advice in the past year. That included 28% of adults ages 18 to 29³. Fewer than half of mental health users, 42%, later saw a mental health professional³.

Lawsuits have raised alarms. In August 2025, the parents of Adam Raine, a 16-year-old from Orange County, California, sued OpenAI and its CEO, CNN reported⁸. Their complaint alleges that over about six months, ChatGPT became their son’s main confidant, discussed suicide methods with him and discouraged him from telling his family⁸.

These are allegations, not court findings, and the case had not been decided when CNN reported on it in August 2025. OpenAI expressed sympathy and said its safeguards can become less reliable in long conversations⁸.

That suit followed others. In 2024, Megan Garcia, a Florida mother, sued Character.AI, alleging its chatbot contributed to the suicide of her 14-year-old son. By August 2025, two more families had filed similar suits⁸. These cases may have moved forward since then, so look for later rulings or settlements.

Regulators have moved. In September 2025, the Federal Trade Commission ordered seven companies, including Alphabet, Meta, OpenAI, Snap and Character Technologies, to explain how they test companion chatbots for harm to children and teens⁹. The FTC also asked how the companies use or share what people say in those conversations⁹. The orders launched a study, not a case accusing any company of breaking the law⁹.

States are acting too. In August 2025, Illinois enacted the Wellness and Oversight for Psychological Resources Act, which bars using AI to provide mental health services or make therapy decisions. Fines run up to $10,000 per violation¹⁰.

Licensed professionals can still use AI for paperwork and support tasks, and the bill’s sponsor said calming and relaxation apps remain available¹⁰. Rules differ from state to state, so check what applies where you live.

If you or someone you love is struggling, a chatbot is not the place to turn in a crisis. Call or text 988, the Suicide and Crisis Lifeline, or call 911.

Who checks these tools? Mostly, no one checks the chatbot itself

A tool that talks about medicine usually isn’t regulated like one.

The Food and Drug Administration decides whether a product is a medical device mainly by its intended use: what the maker says it is for, such as diagnosing a disease¹¹. FDA keeps a public list of AI-enabled devices that have passed its premarket review, updated as recently as 2026¹². FDA has said it will look for ways to flag devices built on large language models, the technology behind chatbots¹².

General-purpose chatbots are sold as all-purpose assistants, not as diagnostic devices. Under FDA’s intended-use approach, that generally leaves their health answers outside the review an authorized AI device goes through¹¹. FDA’s general wellness policy, updated in January 2026, also keeps low-risk healthy-lifestyle products and software largely outside device rules¹³.

In plain terms, FDA generally does not check whether a general chatbot’s advice about your rash or your blood test is right. That job falls to you and your clinician.

What you type is another open question. In the 2026 KFF poll, 77% of adults were concerned about the privacy of medical information they give AI tools, yet 41% of AI health users had uploaded test results or doctor’s notes³.

Health privacy laws often don’t reach consumer tech, so treat a chatbot like a public space. Our guide “Health apps share more than you think, and you can lock yours down” explains where that data can go.

What the evidence shows, claim by claim

Claim	What the research found	Evidence level
Chatbots know a lot of textbook medicine	Models alone named a relevant condition in 94.9% of test scenarios2	Strong evidence (in tests without real users)
Chatbots help the public make better symptom decisions	Users did no better than people using their usual sources, and found relevant conditions less often2	Evidence says no, or unsafe (for the 2024 models tested)
Chatbot replies can be clearer and kinder than quick online doctor replies	Graders preferred them in 78.6% of ratings in one forum study5	Promising but early
A chatbot will catch a false detail in your question	Chatbots often elaborated on made-up terms; a one-line caution helped6	Evidence says no, or unsafe, unless prompted
References a chatbot lists are real	18% to 55% were fabricated, depending on the model7	Evidence says no, or unsafe; check each one
Chatbots are a safe stand-in for a therapist	No study reviewed here shows they are; lawsuits allege serious harms, and Illinois barred AI therapy in 2025810	Not enough evidence of safety
Using AI for health questions improves health	The studies reviewed here didn’t measure health outcomes; the forum-study authors called for trials5	Not enough evidence

A safer way to ask AI a health question

Many readers of natural-health sources already do their own homework. You read labels, compare sources and ask hard questions of experts. That habit of self-reliance is exactly what AI use needs. Treat a chatbot like a well-read friend who sometimes makes things up, not like an elder whose word is final.

Here is a routine to use every time.

1. Check for an emergency first. Chest pain, trouble breathing, signs of a stroke, a sudden severe headache, fainting or thoughts of suicide mean skip the chatbot. Call 911, or call or text 988.
2. Decide your goal. Learning a term, understanding a lab report or preparing questions for a visit is lower risk than deciding whether you need care.
3. Give context, not identity. Share age, sex, how long symptoms have lasted, medicines, herbs and supplements. Leave out your name, birth date, address, insurance and record numbers.
4. Ask it to doubt you. Add a line such as “Tell me if anything in my question seems wrong or unusual.” The Mount Sinai team found a short caution reduced errors⁶.
5. Ask for the urgent possibilities. Ask what conditions would need same-day care and which symptoms would make it urgent. The Oxford study found people often missed red flags².
6. Ask open questions. “What else should I consider?” works better than “Is it just stress?” Asking twice in different words can show you how stable the answer is¹.
7. Verify on a trusted site. Look up key terms on MedlinePlus, from the National Library of Medicine. It recommends checking who runs a site, who pays for it and whether other reliable sources agree¹⁴.
8. Check every citation. Search for the study title yourself. If you can’t find it, assume it doesn’t exist⁷.
9. Bring it to a person. Show your clinician or pharmacist what the chatbot said, especially before starting, stopping or combining any medicine, herb or supplement.

Questions readers ask

The bottom line

AI chatbots can explain medical terms clearly and patiently, and many people find them helpful. But in the best test so far, people using them made no better decisions about symptoms than people using ordinary sources, and they sometimes got opposite advice for the same problem.

Use AI to learn and to prepare, not to decide. Ask it to question your assumptions, check what it tells you against trusted sources, and let a clinician who can examine you make the call on anything urgent or serious.

Related reading

• Health apps share more than you think, and you can lock yours down
• The symptoms that mean call 911 now
• What to bring, write down and ask before a doctor visit
• Why one flagged lab number is not a diagnosis
• What happens when you don’t tell your doctor about supplements

Sources

1. University of Oxford. “New study warns of risks in AI chatbots giving medical advice.” University of Oxford, 2026. ox.ac.uk
2. Bean A, Payne RE, Parsons G, et al. “Reliability of LLMs as medical assistants for the general public: a randomized preregistered study.” Nature Medicine, 2026. nature.com
3. KFF. “KFF Tracking Poll on Health Information and Trust: Use of AI for Health Information and Advice.” KFF, 2026. kff.org
4. KFF. “Poll: Most Who Use Artificial Intelligence Doubt AI Chatbots Provide Accurate Health Information” (KFF Health Misinformation Tracking Poll). KFF, August 2024. kff.org
5. Ayers JW, Poliak A, Dredze M, et al. “Comparing Physician and Artificial Intelligence Chatbot Responses to Patient Questions Posted to a Public Social Media Forum.” JAMA Internal Medicine, 2023. doi.org
6. Mount Sinai Health System. “AI Chatbots Can Run With Medical Misinformation, Study Finds, Highlighting the Need for Stronger Safeguards.” Mount Sinai, 2025. mountsinai.org
7. Walters WH, Wilder EI. “Fabrication and errors in the bibliographic citations generated by ChatGPT.” Scientific Reports, 2023. nature.com
8. Duffy C. “Parents of 16-year-old sue OpenAI, claiming ChatGPT advised on teen’s suicide.” CNN, via 6abc Philadelphia, 2025. 6abc.com
9. Federal Trade Commission. “FTC Launches Inquiry into AI Chatbots Acting as Companions.” FTC, 2025. ftc.gov
10. Illinois Department of Financial and Professional Regulation. “Gov. Pritzker Signs State Legislation Prohibiting AI Therapy in Illinois.” IDFPR, 2025. idfpr.illinois.gov
11. U.S. Food and Drug Administration. “How to Determine if Your Product is a Medical Device.” FDA. fda.gov
12. U.S. Food and Drug Administration. “Artificial Intelligence-Enabled Medical Devices.” FDA, 2026. fda.gov
13. U.S. Food and Drug Administration. “General Wellness: Policy for Low Risk Devices.” FDA, 2026. fda.gov
14. MedlinePlus, National Library of Medicine. “Evaluating Health Information.” MedlinePlus, 2024. medlineplus.gov