🔍 Read the full analysis: A 1-In-5 Pass Rate, A Near-Top Rank: Z.ai’s Legal AI Explained on ThorstenMeyerAI.com
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
Z.ai’s GLM-5.3-FlashX-Legal scored 19.17% on Vals AI’s leaderboard for Harvey’s Legal Agent Benchmark, placing seventh among 81 models in the archived results. The score requires a model to pass every rubric criterion on a task; Vals separately reports a 92.5% pass rate across individual criteria. Those results do not establish that the model is safe to use without human review.
Z.ai’s GLM-5.3-FlashX-Legal scored 19.17% on Vals AI’s leaderboard for Harvey’s Legal Agent Benchmark, ranking seventh among 81 models in the archived results. The score counts a task as passed only when every required criterion is met, so the result indicates a complete pass on about one in five tested tasks—not that the model got only one-fifth of its answers right.
The score is listed on Vals AI’s archived Version 1 leaderboard, updated October 18, 2026. Vals reports an additional 92.5% pass rate on individual criteria for GLM Legal. These metrics describe different levels of grading: a task can miss one required deadline, citation or other item and fail the all-or-nothing task measure while still meeting most of its individual criteria.
Z.ai’s developer documentation describes the model as optimized for professional legal workflows, including document understanding, legal reasoning, research, contract review and drafting. The company says the model underwent legal-domain mid-training and post-training, including reinforcement learning in tens of thousands of simulated legal-workflow environments. Z.ai recommends using it as an agent connected to services providing laws, cases and company data.
The leaderboard places GLM Legal at $0.15 per test, below the listed costs for higher-ranked models. Z.ai’s pricing page separately lists token rates of $0.37 per million input tokens and $1.25 per million output tokens. These are reported pricing figures, not an estimate of total deployment costs, which can depend on usage and connected services.
A High Criteria Score, Few Complete Tasks
The gap between 92.5% of criteria passed and 19.17% of tasks completed shows why benchmark definitions matter for legal work. A work product may contain many correct elements and still fail the task if it misses one required item. Harvey’s benchmark is designed to make that distinction visible rather than treating partial completion as a full pass.
For law firms and other organizations, the result offers a comparison point, not a deployment recommendation. Legal documents can turn on specific facts, deadlines and citations, so strong average performance across rubric items does not establish that an individual output is reliable. Buyers still need to examine verification procedures, data handling and human review, including who processes client information and what checks occur before work is used or shared.
The price-per-test listing may interest organizations comparing evaluation costs, but it does not by itself show the cost or value of using the model in production. Vals’ leaderboard measures performance on a defined set of tasks, while actual legal workflows may involve different documents, jurisdictions, confidentiality requirements and consequences for errors.
Products Worth Considering
Bose QuietComfort Headphones (2nd Gen), Wireless Over-Ear Noise-Cancelling Headphones, Hazelnut Taupe
DJI Osmo Mobile 8 3-Axis Gimbal Stabilizer for Phones
DJI Osmo Mobile 8 Smartphone Gimbal with Mic Mini Transmitter, Infinity Black
As an affiliate, we earn on qualifying purchases.
How Harvey Grades Legal Agents
Harvey released its Legal Agent Benchmark (LAB) as an open-source benchmark on May 6, 2026. It contains 1,250 tasks across 24 practice areas. Each task gives an instruction and client matter materials, then asks an AI agent to produce a work product. More than 75,000 expert-written rubric criteria assess details such as facts, conclusions, citations, recommendations, deadlines, amounts and formatting.
Under Harvey’s all-pass grading rule, a task counts as complete only if every criterion passes. Vals AI’s public leaderboard uses a private holdout set of 120 tasks, with internet access disabled. According to Vals, two AI judges grade submissions and the reported score averages their task pass rates. That makes the 19.17% a benchmark result under a specific test and grading setup, not a general measure of legal competence.
Z.ai’s October 13 AutoClaw blog post announced an application-based beta and described the model as built on GLM-5.3-Flash. Vals lists the release date as October 12. Z.ai claims the legal model improves by 12.5 percentage points over the base model and achieves state-of-the-art performance among open-source models. On the archived leaderboard, the listed base-model score is 6.67%, making the stated point difference consistent with the figures. The source material does not establish the comparison set behind the open-source superlative.
“A model optimized for professional legal workflows, with a focus on document understanding, legal reasoning, and task execution.”
— Z.ai, in its developer documentation
Limits of the Benchmark Evidence
The leaderboard provides a result for a 120-task private holdout, but the source material does not provide enough detail to assess how closely those tasks match any particular firm’s work. The result also depends on the benchmark’s rubric and two AI judges. It does not show performance across every jurisdiction, document type or live client matter.
The source material reports a four-task spot check by Thorsten Meyer in which the legal model cited rulings that could not be found. That small, non-benchmark check cannot establish a general error rate. The same account says OpenAI and Claude models in the check invented no rulings, while two Claude models cited a real ruling for the opposite of what it held. These observations are limited to the specific tasks described.
It is also unclear from the cited material what the company means by “open-source” in its comparison claim; the author reports finding no published weights for the legal model. Z.ai’s documentation page has no publication date, and the source material does not specify who processes client data, what contractual protections apply, or whether the model’s output carries a legal-advice disclaimer. Z.ai describes outputs as drafts for legal professionals.
Beta Access and Buyer Checks
Z.ai’s AutoClaw post says access to the model’s beta is by application. The company’s documentation recommends connecting it to legal and company-data services through MCP, but the cited material does not give a wider release schedule or details about availability in particular markets.
Organizations considering a trial will need to verify the model’s performance on their own representative tasks and establish review requirements for citations, deadlines and other consequential details. They will also need answers about client-data processing, retention and applicable contractual or regulatory terms. No further benchmark release, updated leaderboard result or deployment milestone is confirmed in the source material.
Key Questions
What does the 19.17% score mean?
It is the share of benchmark tasks on which every required rubric criterion passed, according to Vals AI’s archived leaderboard. A task with one or more failed criteria does not count as a full pass.
How can the model pass 92.5% of criteria but only 19.17% of tasks?
The figures use different measures. The 92.5% figure counts individual criteria, while the task score requires every criterion within a task to pass. Missing even one item can fail the whole task.
Did GLM-5.3-FlashX-Legal rank first?
No. On the archived Vals AI Version 1 leaderboard cited here, it ranked seventh among 81 models. The ranking reflects that leaderboard’s benchmark and version, not every possible legal-AI comparison.
Does this score show the model is ready to handle legal work on its own?
No. The benchmark result does not establish suitability for independent legal work. Organizations would need to test the model on relevant tasks and use qualified human review, especially for citations, deadlines and client-facing work.
What is known about access to the model?
Z.ai’s October 13 AutoClaw blog post described an application-based beta. The source material does not give a general availability date or full details about client-data handling.
Source: ThorstenMeyerAI.com
