Microsoft’s AI Study: The Results Depend On What Was Measured

🔍 Read the full analysis: Microsoft’s AI Study: The Results Depend On What Was Measured on ThorstenMeyerAI.com

Before you orderOffer from Amazon

Get the latest gadgets delivered free with Prime

• Fast, free delivery on millions of items
• Prime Video, Amazon Music and more included
• Member-only deals all year

Start your free Prime trial →Free trial for eligible customers · Cancel anytime

As an affiliate, we earn on qualifying purchases.Hide

Microsoft’s MAI-DxO scored 83.2% on 304 difficult medical cases in a paper published October 22 in the peer-reviewed journal Science, compared with 36.1% for generalist physicians using textbooks and the internet and 22.3% without those resources. The result is a benchmark performance, not evidence that the system improves diagnosis or outcomes for real patients: the test used historical text cases and compared a multi-agent AI system with doctors working alone.

The paper, “Sequential diagnosis with language models,” describes Microsoft’s Sequential Diagnosis Benchmark, designed to test how a system gathers information as well as whether it reaches a final diagnosis. Each case begins with a short summary. The model or physician must request additional details, such as questions or tests, from a gatekeeper holding the case file. The benchmark assigns costs to tests and evaluates both diagnosis and resource use.

Microsoft reports that MAI-DxO reached 83.2% accuracy on 304 New England Journal of Medicine cases. On a separate set of 325 emergency department cases, the company reports scores of 80.3% for MAI-DxO and 46.8% for physicians. In the complex-case comparison, physician performance was 36.1% with reference materials and 22.3% without them. These are results reported by the study’s authors, all of whom are affiliated with Microsoft; publication in a peer-reviewed journal does not mean independent researchers have reproduced them.

The system uses a language model in a panel-like process: virtual agents propose diagnoses, request the next piece of information, and challenge one another over multiple rounds. Microsoft’s chart also reports that GPT-5 alone scored 78.9% on the complex cases, close to MAI-DxO’s result. The paper attributes roughly four additional percentage points to the orchestration approach; it also estimates lower test costs for the system, though those costs are modeled rather than actual clinical spending.

What the Scores Say About Diagnosis

The study matters because it tests a more realistic diagnostic task than simply selecting an answer after every detail has been provided: deciding what information to seek next, and whether the expected value of a test justifies its cost. That could be relevant to future decision-support systems, where asking better questions and avoiding unnecessary tests may be as important as naming a condition.

But the reported gap should not be read as evidence that MAI-DxO is twice as effective as doctors in hospitals. The physician comparison was against primary-care doctors and hospital generalists working individually, while the AI system used multiple agents. The cases were deliberately challenging teaching examples, not a representative stream of patients. The study did not test whether using the system changes treatment, reduces errors, improves health, or saves money in routine care.

The distinction has practical consequences for hospitals, insurers and patients. Accuracy on a curated case set can support further research, but clinical adoption also depends on safety, workflow, accountability, performance across patient groups, and evidence from real-world use. The benchmark’s cost estimates cannot by themselves show that a hospital would spend less.

How Microsoft Built the Benchmark

In conventional medical-question benchmarks, systems receive a set of facts and choose a diagnosis. Microsoft’s Sequential Diagnosis Benchmark instead makes participants request information in stages. The design aims to resemble the sequence of history-taking and test selection in diagnosis, while allowing the researchers to score the final answer and the information-gathering path.

Details described in the source account draw partly on the June 2025 arXiv preprint, which was not peer-reviewed; those methods may differ from the final Science paper. In that version, a model judged answers against the published diagnosis, and physician re-grading showed good but imperfect agreement. If a requested test result was absent from a case record, a gatekeeper model generated a synthetic result. Test prices were estimated using billing codes and a 2023 U.S. hospital price list, with a fixed physician-visit cost. The authors described these prices as first-order approximations.

The comparison also changed between versions. The earlier preprint reported an AI score of 80% against 20% for doctors, using an earlier model, OpenAI’s o3. The peer-reviewed study reports the newer figures and includes the open-book physician comparison. In the preprint, the doctors were 21 U.S. and UK generalists, and the AI and physicians were not evaluated on identical numbers of cases. Readers should rely on the final paper for its reported findings while treating preprint-specific methods as provisional where the final text is not available.

“I really believe that in 2027, we will be on the verge of a coding style breakthrough for medical records.”

— Microsoft CEO of AI Mustafa Suleyman, speaking to TechCrunch

Limits of the Clinical Evidence

The study does not establish how MAI-DxO performs with real patients, physical examinations, imaging, or live clinical workflows. The cases were historical and text-based, and the comparison did not include specialists, physician teams, or colleagues collaborating on a diagnosis. The study also does not report patient outcomes from using the system in care.

The case set creates further limits. According to the source account, the NEJM cases contained no healthy patients or harmless explanations, so the benchmark could not measure false alarms in the way a broader patient population might. A system tested mainly on difficult cases may behave differently when faced with common conditions and patients who do not have a serious disease. It is also unclear from the available account how well the results generalize across hospitals, patient populations, and medical specialties.

Peer review means the paper underwent journal review; it does not resolve those questions or provide independent replication. The final paper’s full text is behind a paywall, and some methodological details available in the source come from the earlier preprint. The reported figures should therefore be understood as study results under specified test conditions, not settled estimates of clinical performance.

Testing MAI-DxO With Patients

The next stated step is a prospective study at Beth Israel Deaconess Medical Center in Boston. According to the source account, consenting patients are expected to interact with the system alongside their normal care. Such a study could provide evidence about how the tool operates in a clinical setting, but the source does not provide a start date, enrollment target, protocol, or planned outcome measures.

Readers should watch for whether future reports describe the study design and results in enough detail to assess safety, physician oversight, diagnostic errors, patient outcomes, and effects on testing or costs. Until those data are available, MAI-DxO’s high benchmark scores remain evidence about performance on the study’s case sets—not proof that it is ready to replace or outperform clinical teams.

Key Questions

What did Microsoft’s study find?

Microsoft reports that MAI-DxO scored 83.2% on 304 difficult NEJM cases. Generalist physicians scored 36.1% with textbooks and the internet and 22.3% without them. A separate emergency-case comparison reported 80.3% for the system and 46.8% for physicians.

Does the study show that MAI-DxO is better than doctors in hospitals?

No. It compared a multi-agent AI system with generalist doctors working alone on historical, text-based cases. It did not test MAI-DxO against specialist teams or measure patient outcomes in routine care.

How much did the orchestration system add?

The chart cited in the source reports 78.9% for GPT-5 alone on the complex cases, compared with 83.2% for MAI-DxO. The source describes the system’s additional gain as about four percentage points. Those are benchmark scores, not clinical outcome measures.

What happens next?

A prospective study is planned at Beth Israel Deaconess Medical Center, where consenting patients are expected to interact with MAI-DxO alongside their usual care. Details such as the timetable and study measures were not provided in the source material.

Source: ThorstenMeyerAI.com