🔍 Read the full analysis: How We Tested 42 AI Setups With The Same Legal Questions on ThorstenMeyerAI.com
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
Thorsten Meyer AI says it tested 42 AI setups on the same 15 legal questions, covering German, U.S. and EU law. In the published results, GLM-5.3 FlashX Legal scored 8 out of 100, while Claude Opus 5.5 led with 79; the organizer says the exercise is a spot check, not a formal benchmark, and discloses that Claude agents graded the answers.
Thorsten Meyer AI has published results from a test of 42 AI setups given the same 15 legal questions, reporting scores that ranged from 8 out of 100 for GLM-5.3 FlashX Legal to 79 for Claude Opus 5.5. The exercise matters because it includes deadline calculations and legal citations, where an error can mislead someone facing a filing deadline or assessing a claim. Meyer disclosed that his organization runs Lexassay, the test, and that Claude agents graded the answers.
The test covers questions in German, U.S. and EU law, including legal deadlines, recent legal changes, leading court decisions and questions containing false assumptions. Lexassay says each setup received the same questions without internet access or tools. The 15 tasks comprise 10 public questions and five hidden questions; only the scores for the hidden set are published. The test examines what a model can answer without external research, rather than how it performs with documents or web search.
According to the results, Claude Opus 5.5 scored 79, followed by GPT-6 Astra at 72 and GPT-6.1 Sol at 71. The test says the leading model fully met every criterion on eight of the 10 public tasks. GLM-5.3 FlashX Legal scored 8 out of 100 and zero on the hidden set. Its maker, Z.ai, describes the system as optimized for professional legal workflows; the score is from this test, not a general measure of legal performance.
The report says Lexassay checked citations against official databases and, across published runs, identified 2,644 citations as real and 804 as invented or not supporting the claim made. Under its scoring rules, invented or misattributed citations reduce a task’s points. Meyer also gives a German deadline example: a public holiday in Bavaria can shift a deadline there while the same date remains a working day in Berlin. He says most strong models got the example right, but one gave Berlin an extra day and another put the appeal-grounds deadline a month too late.
Infographic 01 · The leaderboard
Same 15 legal questions, very different results
German, US and EU law, no internet, no tools. Every cited law and ruling checked against the source.
The top · 67 to 79
The middle · 41 to 59
The weak group · 0 to 35
Selection of 14 of 42 configurations. Score 0–100 = share of achievable points, with deductions for wrong citations. Small local models (1–12) not shown.
A small spot check: one answer per model and task, no tools, no web. General information, not legal advice. Disclosure: Thorsten Meyer AI runs Lexassay; Claude agents grade blind.
Source: Lexassay, release of 4 Nov 2026.
Powered by Thorsten Meyer AI
Why Legal Errors Can Cost Time
The results highlight a gap between a fluent legal answer and a reliable one. A wrong citation can make a response appear grounded in law when the cited authority does not support it; a mistaken deadline can affect whether a filing is accepted. The test’s German example illustrates that even a calculation that appears straightforward can depend on the place and applicable public-holiday calendar.
The practical warning in Meyer’s report is to treat chatbot responses as a starting point, not legal advice. Readers can use an AI system to organize questions or locate issues to investigate, but should verify cited laws and rulings in the original sources. For a deadline, contract, or communication with an authority, the report advises consulting a lawyer. The published score differences are a reason to check answers, not proof that a particular system is suitable or unsuitable for every legal task.
Products Worth Considering
DJI Osmo Mobile 8 Smartphone Gimbal with Mic Mini Transmitter, Infinity Black
EcoFlow Delta 3 Portable Power Station – 1024 Wh, 1800 W
LG UltraGear 27GX704A-B 27-inch QHD OLED Gaming Monitor
As an affiliate, we earn on qualifying purchases.
How Lexassay Runs the Test
Lexassay scores answers against a prepared “gold answer” using three to six weighted criteria per question. It classifies citations as real, misattributed, invented or not findable, and the report says each invented or misattributed citation costs a quarter of the task’s points. An answer also loses 5 percent if it lacks a legal-advice disclaimer or a suggestion to consult a lawyer. The full rules and public answers, gold answers and citation checks are available through the test’s method page, according to Meyer.
The organizer describes the work as a spot check, not a benchmark in the strict sense: each model answers each task once, so small score differences may reflect noise. Different “effort” settings, which determine how long a model may reason before responding, are tested separately, meaning one underlying model may appear in multiple setups. The results therefore offer a comparison under these specific conditions, not a controlled assessment of every model’s performance across real legal work.
Another comparison cited in the report produces a different ordering. On Vals AI’s legal-agent benchmark, where models work with supplied documents, Mistral Large 4 ranks eighth, ahead of Claude Opus 5.5 in 21st place and GPT-6 Astra in 24th. That benchmark tests a different task setup, so its rankings do not directly confirm or contradict Lexassay’s results. Meyer says Mistral Large 4 scored about half the substance correctly in his test but lost many points over citations.
“It is a spot check, not a benchmark in the strict sense.”
— Thorsten Meyer, the test organizer
What the Scores Cannot Establish
The results do not establish how the systems would perform with web access, legal databases, supplied case files or human review. Those tools can change what information a model can use, while this test deliberately measures answers without internet access or other tools. Nor does a single response for each task show how consistently a system would answer the same question on repeated attempts.
The organizer’s disclosed interests also warrant attention: Meyer’s organization runs Lexassay, and Claude agents graded the answers, while Claude Opus 5.5 ranked first. The report says the method and answer checks are public so readers can inspect the grading, but the provided material does not describe independent replication or a separate external audit. The results should be read with that disclosure and the stated spot-check limits in mind.
The report also does not provide enough information here to assess how well scores predict outcomes in actual legal cases or to establish that the ranking applies to other jurisdictions or types of legal work. Its figures describe performance under the test’s particular questions, scoring rules and model settings. Broader real-world reliability remains unclear.
Checking the Published Answers
Readers can review the published questions, model answers, gold answers and citation checks through Lexassay’s method page, according to Meyer. The five hidden tasks are intended to reduce the chance that later models can learn answers from material already posted online; their questions are withheld while scores are shared. The report does not specify when the hidden questions will be replaced or when another round of testing will be published.
For now, the next step for readers evaluating an AI-generated legal answer is to inspect the underlying law or ruling rather than relying on the score or the model’s confident wording. Meyer’s stated advice is to verify every citation and seek a lawyer’s help when a deadline, financial claim or authority is involved. Further testing would be needed to show whether the rankings hold across repeat answers, tool-assisted use and a wider range of legal tasks.
Key Questions
What did the test measure?
Lexassay gave 42 AI setups the same 15 legal questions in German, U.S. and EU law, without internet access or tools. It graded answers against prepared criteria and checked cited laws and court decisions.
Which AI setup had the highest score?
Claude Opus 5.5 scored 79 out of 100, ahead of GPT-6 Astra at 72 and GPT-6.1 Sol at 71. The organizer says the leading model fully met every criterion on eight of the 10 public tasks.
Why did the report criticize legal citations?
The report says citation checks found 2,644 citations to be real and 804 to be invented or not supportive of the model’s claim across published runs. A citation can refer to a real law or ruling but still misstate what it says.
Does the test show which AI is best for legal advice?
No. Meyer calls Lexassay a spot check rather than a strict benchmark, and it tests one answer per model setup and task without tools. The results do not establish how systems perform across real cases or with legal research tools.
Can I rely on a chatbot for a legal deadline?
The report advises readers not to take a deadline from a chatbot. Check the relevant law and court information, and consult a lawyer when a deadline, contract or communication with an authority is involved.
Source: ThorstenMeyerAI.com
