Wait, so this not an open weight model, correct? Only available via z.ai?
So, if a US or EU person shares their private legal info here, it's processed and stored by an adversarial country that has no extradition treaty with their country?
NATO allies can't even get US soldiers to trial. In Romania we've had cases of manslaughter and the US just shipped them back ASAP and at worst they were discharged dishonorably.
Not sure how good the Harvey benchmark is. I recently looked into traces from an agent solving their firm knowledge task subset and found multiple problematic tasks and rubrics.
When I first tried using IDE AI coding assistants a bit over a year ago, they would frequently fail to read files and completely hallucinate the code I asked it to analyze or edit. And even when it did successfully read my code, it would often hallucinate the API of other frameworks.
I don’t think hallucinations are a major issue with agentic coding a year later? At least, not for me.
Legal writing does have the problem that there is no such thing as a correct argument that a goal-driven loop can target. But basic things like “does this case even exist” seem solvable with research agents, and even harder problems like “does this case support the argument being made” seem conceptually very similar to questions I now ask Claude daily.
An issue I come across is any attributed failures of a first time use of AI.
Non deterministic software can behave orders of magnitude different based on how input is structured.
This means the same problem can work for one person reliably, and not another and both are true.
Assertions that begin with a deterministic assessment using a snapshot in time of both the user and models capabilities are a new feature of this new category of software that will have to be better caught and expressed.
I provided some very light guidance to a couple of masters students who were working on legal citation hallucination detection last summer. Their summer project is here: https://github.com/gt-csse/mellea-lrc
If anyone here is looking for engineers to work on legal AI products, both of the students who worked on this project come highly recommended.
To your second point, it seems like every time a significant practical limitation is identified, it goes through a familiar pattern. It gets a news cycle, and then it gets a wave of confidently incorrect opinionating, and the problem gets treated as if it's a permanently built in problem that's baked into the foundation of AI. And then time passes and the issue is either mitigated or circumvented entirely.
Hallucinations are one, "AI can't scale anymore" is another (remember that one?), limited context windows, strawberry problems, glue on pizza, sycophancy.
I don't want to say there's no problems, there are very serious ones. User psychosis, economic harms, alignment, cost of computer components, etc. But there's a lot of false positives that get turned into Ed Zitron articles along the way.
As an attorney, I am not particularly impressed about a 20% pass rate on a benchmark. The name of the game for litigation practice is about being prepared for novel situations, not relitigating past ones.
To be fair to them, on this particular benchmark (Harvey's Legal Agent) 20% puts it almost top of the pack of _any_ model, and certainly top for any remotely in its price class.
How much novelty can there be?
If attorneys were as diligent and generous with “open sourcing” their data as SWEs had been, could LLMs benefit from all that training data?
Law and code are only superficially similar as law always requires human interpretation plus law is an interface to reality. Code can stay contained in its nice, little, abstract box.
What are you talking about? 90% of litigation is convincing the judge that existing case law that clearly applies to your case does apply to your case.
I'm guessing what they're saying is that nothing goes to litigation if it's a clear pattern-match to existing cases, so actual litigation practice is only novel cases.
Yeah agreed, especially for legal and healthcare models.
[edit] I'd go one step further too and say the training data should be explicitly requestable too so third parties can verify (freedom of information act styles).
"Freakazoid has ... access to all of the Internet's knowledge. He cannot fly, but often runs around with his arms extended and making whooshing sounds with his mouth, pretending he is."
Sort of ironic, in Google Chrome, when attempting to visit that legal z.ai site I get:
Your connection is not private
Attackers might be trying to steal your information from z.ai (for example, passwords, messages, or credit cards). Learn more about this warning
net::ERR_CERT_COMMON_NAME_INVALID
Curious what domain is next, GLM-Science? And when will it be profitable to drill down to even more domain specific models like GLM-Economic-Geography or GPT-Aquaculture?
Imo, we are going to see a lot more specialised models in the near future as pushing the frontier gets harder. Qwen 3.8 27B for example is a pretty rubbish chat and general world knowledge model, but is stellar at coding (for it's size). Its small enough anyone with a modest gaming PC (16GB vram and 32GB of DDR5) can run it at speeds good enough for overnight coding sessions with an PR to review at the end of it.
Eventually, I hope we'll see many more specialised small models that run nicely on consumer and prosumer hardware. Like LLMs specific to income tax law for UK residents, or LLMs specific to building power delivery circuits for PD USB C devices. There probably is a point where the niche-ness does become to niche to justify the post training, but again, small models means much less training required.
Kind of begs the question if this even needs new/better data? Could you take the foundation model that's trained on everything, and then run a final fine tuning training run on just the set of foundation training data that you want the model to specialise in? Eg, extract just the training data that is about UK income tax law and then run a few epochos on just that? Any ML people know if this is a thing or if not, why it doesnt work?
A lot of the examples they show seem to be focused on common law countries. The US and UK are where most of the money is to be made in the legal industry, but I am curious to see what ripple effects we get from people attempting to use AI tools geared toward US/UK-style legal systems in other kinds of systems.
The usual concern is attorneys, but judges are probably even more overloaded these days. Can this help judges as well? If so, would this ultimately lead to better decisions?
The legal space understands well how to tier services based on who has the liability. E.g. trainees and paralegals whose work is filtered through the lawyer who ultimately puts their name on what gets submitted to courts.
I'm sure we'll continue to see the occasional person who fails to understand they themselves will be held liable if they submit work done by an AI without vetting it, but it's not like they don't have ways of dealing with this.
There should be laws about disclosure on the part of attorneys about their use of AI. I know they're already using it, but spending years or decades in prison because Lionel Hutz, Esq. did something stupid at a prompt sounds unappealing to me.
I think they should be somehow restricted to using non-generative models (or at least for the user facing output). Something that would only identify potentially relevant case law from an existing set, to accelerate the search process.
Common law in general seems essentially impossible to formalize in such a way to use purely deterministic reasoning models of yesteryear, but I'm sure many specific laws could be converted, especially in fields like tax law (I'm sure they already are by many big players)
A lot of law is combing through large bodies of rule sets and reference cases to find inputs or parallels relevant for a case. Are you sure you'd rather have Lionel Hutz, Esq. with his limited comprehension do that at half a page per minute?
Yes, actually, because Lionel can suffer a consequence. He can be sanctioned. He can be disbarred. He can be sued. He can be thrown in jail.
AI can not meaningfully suffer consequences for what it does, and the trend is to remove the human from the loop more and more.
Furthermore, it's entirely conceivable that you have a law firm staffed by lawyers who more-or-less just say "yeah sure whatever" to what this thing does and, when asked about what can be done to appeal something, or how a document didn't make it into evidence, or how a subpoena wasn't filed, just shrug their shoulders and go "iunno". Vibe-practicing law, basically. You need to have an idea of how they're using AI and in what parts of the case.
If Lionel submits junk from an AI, he can still be sanctioned, disbarred, sued, and thrown in jail, just as he could if he let some incompetent paralegal screw up and he submits the resulting documents without checking them.
It's ultimately his name on the documents submitted to the court.
> but spending years or decades in prison because Lionel Hutz, Esq. did something stupid at a prompt sounds unappealing to me.
In which case would this be the primary blame, and not the legal system itself, or you for actually doing the crime?
I fail to see how its your lawyers fault that you went to jail for a crime you didn't commit over the system as a whole. And I really fail to see how it's the lawyer's sole fault that you went to jail for a crime that you should - in fact - have went to jail for...
There are a lot of problems with the legal system (including with lawyers). This just doesn't seem like the hill to die on.
There are a lot of situations where my lawyer's mistake might cause me to go to jail for a crime I didn't commit. There could be some prejudicial evidence that he failed to get excluded, or exculpatory evidence that he failed to get included, or an incorrect ruling in the trial that wasn't properly preserved for appeal.
One of the nice quirks of USA justice system is that you don't have to be guilty of a crime to go to prison, just that the prosecution has to convince 12 people beyond reasonable doubt that you are guilty.
The sixth is taken seriously to the point that incompetent lawyer could be a valid reason for granting a retrial.
Lawyers are going to start living gilded lives. I don’t see us getting past their professional human sign-off so they are essentially going to be guaranteed, easy, high paying jobs with the LLMs doing all the heavy lifting in the future.
That's already a thing. You have paralegals do most of the grungy work for much less than your hourly rate. You still have to show up to court and talk to customers, and these parts are not going away.
While I agree on the premise – (written*) lawyer activity being reduced to putting signatures under things – I’m not so sure the gilded part is how it’ll actually play out in practice.
It might also become a race to the bottom, since you can’t deliver on quality better than others anymore, you’ll have to put effort towards quantity.
That could mean firms might pressure lawyers to sign as much as possible in a given time, while they’re still personally at risk/responsible for the documents they put out. That means either very hectic reading, and missing details, or no reading at all.
*And like socializer said, the written documents are only one part of the job.
I was about to suggest that AI might teach people to become lawyers for cheap and they can just take the BAR exam and become a lawyer...
But then I looked into it and I see that you actually do have to go to certain schools to become a lawyer. Having the knowledge is not enough, you have to pass through the proper (and expensive) gates as well.
Or... the tools get to the point where we don't need a human for expertise but accountability and the role becomes more about ethics, thereby lowering the bar. (PUN!)
It looks like they're relying on tracked changes and human review, but I didn't see anything addressing something that has frequently made the news: LLMs outright hallucinating cases or holdings.
If the human lawyers don't catch that on review, they're likely to have an angry judge and an AI saying "absolutely, you were right to call me out on that, I shouldn't have hallucinated a non-existent case and made the judge sanction you."
Lawyers, essentially the exorbitant and freeloading self-appointed gatekeepers of the deliberately arcane, unaffordable, inaccessible, and patently unequal legal systems all across the world, will now have even their routine/rote, repetitive, menial work done by cheaper LLMs for cheap.
Hopefully. Giving everyone cheap and equal access to the law cannot be a bad thing--well, it might feel bad to those who no longer have exclusive access to the law.
I think an under-appreciate way AIs might disrupt the world is by giving individuals an agent that will ruthlessly optimize them.
Companies have entire departments that spend all their time ruthlessly optimizing so they can charge an extra fee here, or spin up a new shell company there. Meanwhile, here I am in our capitalist society and I just enjoy relaxing and spending time with my wife and kids instead of maximally exploiting the capital I possess at all times--I'm such a fool.
It will be interesting if we ever integrate AI so fully in our lives that the AI can maximally exploit all our capital. My AI might check prices and I will drive past your store to save 5 bucks at the next. My AI might help me manage 27 LLCs and shell companies of my own so that I too can just disappear in a poof of legal smoke if I ever need to. Etc.
Wait, so this not an open weight model, correct? Only available via z.ai?
So, if a US or EU person shares their private legal info here, it's processed and stored by an adversarial country that has no extradition treaty with their country?
This sounds like a huge feature though? Basically, a digital attorney–client privilege.
A key point of attorney-client privilege is that it's a feature of the legal system itself, not a technical or political hack.
More like attorney-client liability
You should not trust any LLM to give you accurate legal advice. I can’t imagine all the ways that could go wrong.
Like pattern matching sentence algorithms many can reveal concepts and strategies to confirm independently
Good luck to a EU person getting "US AI people" extradited. They aren't even put on trial in the US.
NATO allies can't even get US soldiers to trial. In Romania we've had cases of manslaughter and the US just shipped them back ASAP and at worst they were discharged dishonorably.
[flagged]
[dead]
Not sure how good the Harvey benchmark is. I recently looked into traces from an agent solving their firm knowledge task subset and found multiple problematic tasks and rubrics.
Here are 3 examples (there are more): https://github.com/harveyai/harvey-labs/issues/191
Approximately every benchmark has significant issues.
Of course. That’s why I opened the issue so that the team can improve it.
No discussion on this can avoid mentioning this ongoing disaster...
"AI Hallucination Cases" - https://www.damiencharlotin.com/hallucinations/
When I first tried using IDE AI coding assistants a bit over a year ago, they would frequently fail to read files and completely hallucinate the code I asked it to analyze or edit. And even when it did successfully read my code, it would often hallucinate the API of other frameworks.
I don’t think hallucinations are a major issue with agentic coding a year later? At least, not for me.
Legal writing does have the problem that there is no such thing as a correct argument that a goal-driven loop can target. But basic things like “does this case even exist” seem solvable with research agents, and even harder problems like “does this case support the argument being made” seem conceptually very similar to questions I now ask Claude daily.
An issue I come across is any attributed failures of a first time use of AI.
Non deterministic software can behave orders of magnitude different based on how input is structured.
This means the same problem can work for one person reliably, and not another and both are true.
Assertions that begin with a deterministic assessment using a snapshot in time of both the user and models capabilities are a new feature of this new category of software that will have to be better caught and expressed.
I provided some very light guidance to a couple of masters students who were working on legal citation hallucination detection last summer. Their summer project is here: https://github.com/gt-csse/mellea-lrc
If anyone here is looking for engineers to work on legal AI products, both of the students who worked on this project come highly recommended.
There being multiple cases every single day is nuts.
But also, interesting how the rate of cases peaked and seems to be going down now.
To your second point, it seems like every time a significant practical limitation is identified, it goes through a familiar pattern. It gets a news cycle, and then it gets a wave of confidently incorrect opinionating, and the problem gets treated as if it's a permanently built in problem that's baked into the foundation of AI. And then time passes and the issue is either mitigated or circumvented entirely.
Hallucinations are one, "AI can't scale anymore" is another (remember that one?), limited context windows, strawberry problems, glue on pizza, sycophancy.
I don't want to say there's no problems, there are very serious ones. User psychosis, economic harms, alignment, cost of computer components, etc. But there's a lot of false positives that get turned into Ed Zitron articles along the way.
As an attorney, I am not particularly impressed about a 20% pass rate on a benchmark. The name of the game for litigation practice is about being prepared for novel situations, not relitigating past ones.
To be fair to them, on this particular benchmark (Harvey's Legal Agent) 20% puts it almost top of the pack of _any_ model, and certainly top for any remotely in its price class.
Opus 5.5 scores 4% on it.
How much novelty can there be? If attorneys were as diligent and generous with “open sourcing” their data as SWEs had been, could LLMs benefit from all that training data?
> How much novelty can there be?
Infinite
In the same sense there could be infinite CRUD apps?
Law and code are only superficially similar as law always requires human interpretation plus law is an interface to reality. Code can stay contained in its nice, little, abstract box.
If professional AI can do all the work that isn’t novel, you free up human talent for edge cases and then improve your AI with new information.
How much of your work is truly novel and unique? Not 100%
What are you talking about? 90% of litigation is convincing the judge that existing case law that clearly applies to your case does apply to your case.
90% of litigation is pressuring a settlement.
I'm guessing what they're saying is that nothing goes to litigation if it's a clear pattern-match to existing cases, so actual litigation practice is only novel cases.
Looks like this is not an open model. I'm curious how the sentiment toward chinese models will shift as labs trend toward keeping them proprietary.
Also, I wonder if this is at all related to the recent reports that Harvey has been getting squeezed on margins by the US labs.
Nobody will have any interest if they are not open
Yeah agreed, especially for legal and healthcare models.
[edit] I'd go one step further too and say the training data should be explicitly requestable too so third parties can verify (freedom of information act styles).
[flagged]
If you are not self-hosting on your hardware inside your network, why do you care whether the model is open or not?
https://quoteinvestigator.com/2019/07/30/lawyer/
"he who will be his own Counsellour, shall be sure to have a Fool for his Client."
How much worse the lawyer who subcontracts Freakazoid?
https://en.wikipedia.org/wiki/Freakazoid!
"Freakazoid has ... access to all of the Internet's knowledge. He cannot fly, but often runs around with his arms extended and making whooshing sounds with his mouth, pretending he is."
Sort of ironic, in Google Chrome, when attempting to visit that legal z.ai site I get:
Your connection is not private Attackers might be trying to steal your information from z.ai (for example, passwords, messages, or credit cards). Learn more about this warning net::ERR_CERT_COMMON_NAME_INVALID
If you’re on a corporate device you may not have recently issued certificates in your cert chain yet
Not the case for me.
Curious what domain is next, GLM-Science? And when will it be profitable to drill down to even more domain specific models like GLM-Economic-Geography or GPT-Aquaculture?
Imo, we are going to see a lot more specialised models in the near future as pushing the frontier gets harder. Qwen 3.8 27B for example is a pretty rubbish chat and general world knowledge model, but is stellar at coding (for it's size). Its small enough anyone with a modest gaming PC (16GB vram and 32GB of DDR5) can run it at speeds good enough for overnight coding sessions with an PR to review at the end of it.
Eventually, I hope we'll see many more specialised small models that run nicely on consumer and prosumer hardware. Like LLMs specific to income tax law for UK residents, or LLMs specific to building power delivery circuits for PD USB C devices. There probably is a point where the niche-ness does become to niche to justify the post training, but again, small models means much less training required.
Kind of begs the question if this even needs new/better data? Could you take the foundation model that's trained on everything, and then run a final fine tuning training run on just the set of foundation training data that you want the model to specialise in? Eg, extract just the training data that is about UK income tax law and then run a few epochos on just that? Any ML people know if this is a thing or if not, why it doesnt work?
A lot of the examples they show seem to be focused on common law countries. The US and UK are where most of the money is to be made in the legal industry, but I am curious to see what ripple effects we get from people attempting to use AI tools geared toward US/UK-style legal systems in other kinds of systems.
The usual concern is attorneys, but judges are probably even more overloaded these days. Can this help judges as well? If so, would this ultimately lead to better decisions?
It will lead to bigger fines :-) https://www.damiencharlotin.com/hallucinations/
Until it hallucinates something no one bothers to verify
Judges hallucinate too. We call them bad rulings.
[dead]
Would be nice if it said what legal system it is set up for. US law? Chinese law? European law? All of them? The author seems to assume we'd know.
the liability model of these models is poor, i think progress in that domain should be a heavy focus.
The legal space understands well how to tier services based on who has the liability. E.g. trainees and paralegals whose work is filtered through the lawyer who ultimately puts their name on what gets submitted to courts.
I'm sure we'll continue to see the occasional person who fails to understand they themselves will be held liable if they submit work done by an AI without vetting it, but it's not like they don't have ways of dealing with this.
Professional AI is so interesting to me, legal, medical, engineering, accounting all can benefit.
And the opportunity for a moat of AI that is correct and accountable(legally) is ripe for the picking.
There should be laws about disclosure on the part of attorneys about their use of AI. I know they're already using it, but spending years or decades in prison because Lionel Hutz, Esq. did something stupid at a prompt sounds unappealing to me.
I think they should be somehow restricted to using non-generative models (or at least for the user facing output). Something that would only identify potentially relevant case law from an existing set, to accelerate the search process.
Common law in general seems essentially impossible to formalize in such a way to use purely deterministic reasoning models of yesteryear, but I'm sure many specific laws could be converted, especially in fields like tax law (I'm sure they already are by many big players)
A lot of law is combing through large bodies of rule sets and reference cases to find inputs or parallels relevant for a case. Are you sure you'd rather have Lionel Hutz, Esq. with his limited comprehension do that at half a page per minute?
This capability has already existed for decades thanks to Westlaw and LexisNexis. These are indispensable tools in the lawyer’s kit.
Yes, actually, because Lionel can suffer a consequence. He can be sanctioned. He can be disbarred. He can be sued. He can be thrown in jail.
AI can not meaningfully suffer consequences for what it does, and the trend is to remove the human from the loop more and more.
Furthermore, it's entirely conceivable that you have a law firm staffed by lawyers who more-or-less just say "yeah sure whatever" to what this thing does and, when asked about what can be done to appeal something, or how a document didn't make it into evidence, or how a subpoena wasn't filed, just shrug their shoulders and go "iunno". Vibe-practicing law, basically. You need to have an idea of how they're using AI and in what parts of the case.
If Lionel submits junk from an AI, he can still be sanctioned, disbarred, sued, and thrown in jail, just as he could if he let some incompetent paralegal screw up and he submits the resulting documents without checking them.
It's ultimately his name on the documents submitted to the court.
That same stuff can all still happen tot he lawyer who uses these tools. If he makes an egregious enough mistake, he's liable.
And that's why you want him to have to disclose that he used those tools.
Why do the tools matters? He is equally liable if he makes the mistakes without them.
> but spending years or decades in prison because Lionel Hutz, Esq. did something stupid at a prompt sounds unappealing to me.
In which case would this be the primary blame, and not the legal system itself, or you for actually doing the crime?
I fail to see how its your lawyers fault that you went to jail for a crime you didn't commit over the system as a whole. And I really fail to see how it's the lawyer's sole fault that you went to jail for a crime that you should - in fact - have went to jail for...
There are a lot of problems with the legal system (including with lawyers). This just doesn't seem like the hill to die on.
There are a lot of situations where my lawyer's mistake might cause me to go to jail for a crime I didn't commit. There could be some prejudicial evidence that he failed to get excluded, or exculpatory evidence that he failed to get included, or an incorrect ruling in the trial that wasn't properly preserved for appeal.
> In which case would this be the primary blame, and not the legal system itself, or you for actually doing the crime?
There are plenty of ways a lawyer can screw up procedurally that can drastically impact your case.
One of the nice quirks of USA justice system is that you don't have to be guilty of a crime to go to prison, just that the prosecution has to convince 12 people beyond reasonable doubt that you are guilty.
The sixth is taken seriously to the point that incompetent lawyer could be a valid reason for granting a retrial.
Lawyers are going to start living gilded lives. I don’t see us getting past their professional human sign-off so they are essentially going to be guaranteed, easy, high paying jobs with the LLMs doing all the heavy lifting in the future.
That's already a thing. You have paralegals do most of the grungy work for much less than your hourly rate. You still have to show up to court and talk to customers, and these parts are not going away.
While I agree on the premise – (written*) lawyer activity being reduced to putting signatures under things – I’m not so sure the gilded part is how it’ll actually play out in practice.
It might also become a race to the bottom, since you can’t deliver on quality better than others anymore, you’ll have to put effort towards quantity. That could mean firms might pressure lawyers to sign as much as possible in a given time, while they’re still personally at risk/responsible for the documents they put out. That means either very hectic reading, and missing details, or no reading at all.
*And like socializer said, the written documents are only one part of the job.
I was about to suggest that AI might teach people to become lawyers for cheap and they can just take the BAR exam and become a lawyer...
But then I looked into it and I see that you actually do have to go to certain schools to become a lawyer. Having the knowledge is not enough, you have to pass through the proper (and expensive) gates as well.
I'm not really surprised.
Or... the tools get to the point where we don't need a human for expertise but accountability and the role becomes more about ethics, thereby lowering the bar. (PUN!)
Do you predict the same paralegal crisis people are claiming about junior swe's?
don't most developers do this already? Senior developers. Juniors are having a hard time.
It looks like they're relying on tracked changes and human review, but I didn't see anything addressing something that has frequently made the news: LLMs outright hallucinating cases or holdings.
If the human lawyers don't catch that on review, they're likely to have an angry judge and an AI saying "absolutely, you were right to call me out on that, I shouldn't have hallucinated a non-existent case and made the judge sanction you."
"advancing". Absolutely laughable, we're talking about less than 20% accuracy on benchmarks. 100% useless.
Lawyers, essentially the exorbitant and freeloading self-appointed gatekeepers of the deliberately arcane, unaffordable, inaccessible, and patently unequal legal systems all across the world, will now have even their routine/rote, repetitive, menial work done by cheaper LLMs for cheap.
Hopefully. Giving everyone cheap and equal access to the law cannot be a bad thing--well, it might feel bad to those who no longer have exclusive access to the law.
I think an under-appreciate way AIs might disrupt the world is by giving individuals an agent that will ruthlessly optimize them.
Companies have entire departments that spend all their time ruthlessly optimizing so they can charge an extra fee here, or spin up a new shell company there. Meanwhile, here I am in our capitalist society and I just enjoy relaxing and spending time with my wife and kids instead of maximally exploiting the capital I possess at all times--I'm such a fool.
It will be interesting if we ever integrate AI so fully in our lives that the AI can maximally exploit all our capital. My AI might check prices and I will drive past your store to save 5 bucks at the next. My AI might help me manage 27 LLCs and shell companies of my own so that I too can just disappear in a poof of legal smoke if I ever need to. Etc.