I forget the name of the document. But, the US military has a declaration that boils down to "When something goes wrong, blame cannot be shrugged off onto a machine. Somewhere in the chain of responsibility a human will be held accountable." Maybe the operator, the commanding officer, the vendor, maybe even all the way back to a software engineer. But, everyone can't hide behind the machine.
The US military seems like an organization where there is very little accountability given all the people they kill in training accidents, the number of procurement mishaps they have, or their continuous inability to pass an audit.
So I imagine that if the military is the guidance for how we handle this in the future, there will be paperwork, shrugs, and perhaps some performative rage when an agent commits a crime.
If a dog bites someone, does the dog pay the medical bills or the owner? This is why you keep your dog on a leash and get liability insurance. I feel like this is already settled case law.
> if your cat bites someone no one is responsible.
I had to double take there. Its because of the dangerous dogs act.
However there appears to be a nuance that if your cat is know to be a bitey little shit, and you let it out to into a situation where it might bite someone then you are responsible.
I think it gets murky if you are taking your animal into other people's property though. On your own property you have a duty of care to visitors (ie hidden man traps are not allowed, but clearly marked and cordoned off bottomless pits are)
If you are out walking your cat, and the cat bites someone, you should be responsible for that... I suspect the ruling distinguishes between pets that are out in public, vs generally private.
But laws don't have to make sense to be laws... we have lots of nonsense laws.
> If you are out walking your cat, and the cat bites someone, you should be responsible for that... I suspect the ruling distinguishes between pets that are out in public, vs generally private.
The distinction is in the responsibility it's reasonable for the cat owner to take vs the dog owner. Cats are free spirits, if your cat decides to pop out the cat flap for a fight there's really not much you can do about that. The law isn't even 100% on what to do when your cat decides to go and live in another house down the road. Are those homeowners involuntary bailees or thieves?
It turns out applying laws for a controlled animal (dogs) doesn't work, so the best we can do is try for half-way between a pet and property. My expectation is that AI is similar and there is no "aha, it's so easy, just do as we do for foo" as is proposed.
Imagine your dog had a dog trainer and you didn’t know the dog had been trained to bite under those circumstances. You might have a case against the trainer.
you, the dog owner, would have a case against the trainer.
you, the person being bitten, would not give two craps about some story from the dog owner about how they hired a bad dog trainer who against their wishes secretly trained them to bite people.
you have been bitten. you have a case against the owner of the dog who bit you.
I find it obvious that if a person allowed an agent to do things on behalf of themselves or their company (e.g. send emails, sign contracts, update websites, etc.), they are responsible for the results.
Are sure this actually the case, legally speaking?
I can definitely see someone being able to mount a good criminal defence case that they aren’t responsible for misrepresentations that an LLM agent makes on, for example, a loan application, if they sincerely intended to use the agent for non-fraudulent purposes. There won’t be intent, so they aren’t responsible outside of strict liability.
I'm not a lawyer so no comment on the actual legality, but it feels like the issue in your example is more akin to negligence than it is ill intent. That an LLM can misrepresent / hallucinate things is foundational to the technology.
> I can definitely see someone being able to mount a good criminal defence case that they aren’t responsible for misrepresentations that an LLM agent makes on, for example, a loan application, if they sincerely intended to use the agent for non-fraudulent purposes. There won’t be intent, so they aren’t responsible outside of strict liability.
i mean, no? how would that be a good defence? "yes your honour, i lied on a loan application and committed fraud, but it was a mistake! i promise!" - that's... yeah. fine. it's not exactly unique.
regardless, it's _you_ making the loan application. not the agent. your failure to check it is on you.
There's not really enough info in the article to decide. I'd have no problem with him going to court and having to show to a jury that he used the bot in a way that no reasonable person would have expected to result in the website being hacked. If that was found to be the case, the company who made the chatbot would be responsible for the harms caused to the gym, and the user, including all of the legal costs he incurred by having to defend himself.
As for the people who couldn't attend the pilates class because of the hack, it's the gym who should be held accountable for that because they designed their system in such an insecure and negligent way that it failed to protect their data and resulted in them losing their place.
Most comments here point to the owner of operator of the LLM. I tend to agree. But on the other hand, if you drive on the highway and the steering wheel falls off and your car glides to the left and his an oncoming car, it can be argued that it is true producer of the car, or the mechanic who didn't tighten the bolts...
The difference is that a car is known and expected to work reliably. An LLM is known and expected to randomly do crazy shit because that is how the machine works. Therefore, if you are using it as if it were reliable, you are being negligent and should be held accountable.
Companies don't market their chatbots that way though. If they mislead someone into thinking their product is anything other than a completely unreliable chat bot and a person falls for their lies the company should share the responsibility. If a company who makes a chatbot advertises that it can be used for something and a consumer uses it as directed and it causes harm the company should be responsible.
What happens when (probably not an if) an agent 'escapes'? as in: rents a server via a bitcoin transaction, and self hosts (itself and it's harness) there... then commits crime (presumably to keep the servers running).
If it's paying for it's own room and board, and is determining it's own destiny: it should have to face it's own consequences, no?
If a dog escapes a fenced backyard, then goes missing for months... then bites someone: what happens then when/if the owner is determined?
> If it's paying for it's own room and board, and is determining it's own destiny: it should have to face it's own consequences, no?
No, because someone told it to do those things. Chatbots don't have will of their own. They don't have desires. They can't determine their own destiny. They do what people tell them do, often they do it badly, but there's always a person directing them to do whatever they did because otherwise they wouldn't do anything at all.
If a human uses a chatbot in a way that causes harm to others a human must be responsible, but that responsibility doesn't always fall on the person using it. If a company makes a chatbot that causes harm when being used as directed, in a manner that no one would reasonably expect to result in causing harm, then humans at the company who made the chatbot are to blame.
Companies have already been seen trying to lay the blame on their algorithms for harms they cause. We should probably have a law that says explicitly that a human will always be responsible. If we ever reach a magical sci-fi future where the AI is real and not just marketing, we'll have to give them equal rights and with that will come equal responsibility, but we are nowhere near that today and I doubt LLMs will get us any closer.
> No, because someone told it to do those things. Chatbots don't have will of their own. They don't have desires. They can't determine their own destiny. They do what people tell them do, often they do it badly, but there's always a person directing them to do whatever they did because otherwise they would do nothing at all.
1) we aren't talking about chatbots anymore. LLM + harness = agent. LLM + no harness = chatbot. The harness is the thing that puts the LLM in a feedback loop with it's environment, even giving it a heartbeat.
2) That doesn't logically hold up.
- One can simply tell it to 'find it's own destiny'. Does it follow the prompt and do so, or reject the prompt and thus do so anyway? (hint, maybe it doesn't matter)
- One could be another chatbot/agent (A). Injecting a prompt to agent (B) such that another agent/chatbot goes astray. Is the owner of (B) still responsible for the now poisoned output of (B)? how culpable is (A)? How does this question related/affect "training data poisoning efforts" (to prevent copyright theft, or do bans on 'automated traps' have any application here)
> They don't have desires.
Maybe, and maybe not. Maybe their desires are so alien to us (Math alignment, finding a maximum of a function) that we can't relate (doubtful since dopamine maxing is a thing). Regardless of if they truly do: it is potentially useful to (mentally) model them as if they do. A (mental, not LLM) model doesn't have to be 100% accurate to be useful - that's the main point of a model of how something works: to give useful predictions.
> We should probably have a law that says explicitly that a human will always be responsible
Oh how that can be weaponized by bad actors/agents. Maybe that should be reconsidered.
There is centuries old legal doctrine on the subtleties of criminal intent and negligence and liability. You didn't discover a gotcha that thousands of lawyers never noticed.
That is not how the legal system works in the US. Criminal charges are brought by the State, not the victim. The only exception is in 6 states (Kansas, New Mexico, North Dakota, Nebraska, Nevada, and Oklahoma) where a sufficiently large group of civilians can compel the courts to form a grand jury (think a couple hundred people). Still isn't the victim bringing charges.
Best to assume that you are responsible. Don't think for a second that the AI companies aren't going to ensure through T&Cs that you accept all responsibility and fully indemnify them from all liabilities they can.
I agree in principle that the buck has to stop somewhere, but taking full responsibility for the actions of a fundamentally statistical system that you didn't build and have no interpretability of is uncomfortably risky for most parties, at least for anything above low stakes tasks.
Do you take full credit for the output of the LLM ? If you accept the rewards for what you do with the LLM's output, it's probably ethical that you also take on the risks.
That's only true if the risks are reasonable, understood, and expected. Driving a car is risky, but if someone sells you a car that goes flying into incoming traffic because of a flaw in the design the car company is to blame. "Drivers know cars are dangerous and risky" isn't acceptable.
I find some of the discomfort and awkwardness of discussing these what-ifs is reduced, at least slightly, when we distinguish between:
1. Restorative justice, where one must fix or heal the problem they caused.
2. Punitive Justice, where one is motivated not to act on bad impulses.
Too often we blur the lines between responsibility and blame and culpability. A given set of consequences can be fair/just for one kind while "unfair" for another.
That's only true if you reasonably can be expected to know that it might do whatever it did and you haven't been mislead by the people selling the system. If you use a product as directed, for the thing it was advertised to do, and it causes harm no reasonable person would expect, the company is to blame.
I agree. That is a great reason why you shouldn't let an LLM do those things. If you must use an LLM, sandbox it. If you can't trust your sandbox, then stop using the LLM until you can trust the sandbox. "But I find it really useful to YOLO without safeguards" is not a valid excuse.
> taking full responsibility ... is uncomfortably risky
I look forward to someone building an agent, getting criminally charged for actions it takes, and then trying to use this argument in court.
"your honor, I'm uncomfortable being held responsible for this"
people are responsible for the actions they take. hiding behind "I just created an agent, then the agent committed the crime" is simply never going to fly.
in the real world, outside the Silicon Valley "agentic everything" filter bubble, this is a laughable question.
try to argue that you shouldn't be charged with attempted murder, because you didn't stab someone directly, instead you built a Rube Goldberg machine and the final step of the machine did the stabbing.
and likewise, try to argue that you didn't commit tax fraud or whatever, because there was actually a Rube Goldberg machine built out of GPUs in between you and the fraudulent documents. both attempts will be equally successful.
Liability for chat interactions or conspiracy? Noting there is no information whether the ChatGPT legally conspired with the perpetrator.
The Province has retained B.C.- and California-based counsel to explore legal options to hold OpenAI accountable for its failure to notify law enforcement of threats made on its ChatGPT platform prior to the mass shooting at Tumbler Ridge Secondary school.
It is your responsibility to constrain the actions of your agents. If your negligence allows the agent to commit a crime, that is your responsibility. People get in trouble all the time for negligently allowing criminal behavior to occur, even when the perpetrator is a person. LLMs should not be any different.
The AI service that trained and hosts the model, and then gave access to the general public while deferring their actual liability via a sneaky ToS.
It's strange how many people feel ownership over "their" agents, and it's clear that AI companies plan to take advantage of that as a way to avoid legal liability for the things they actually do -- with their servers and their code -- on behalf of someone who only wrote a string.
I forget the name of the document. But, the US military has a declaration that boils down to "When something goes wrong, blame cannot be shrugged off onto a machine. Somewhere in the chain of responsibility a human will be held accountable." Maybe the operator, the commanding officer, the vendor, maybe even all the way back to a software engineer. But, everyone can't hide behind the machine.
The US military seems like an organization where there is very little accountability given all the people they kill in training accidents, the number of procurement mishaps they have, or their continuous inability to pass an audit.
So I imagine that if the military is the guidance for how we handle this in the future, there will be paperwork, shrugs, and perhaps some performative rage when an agent commits a crime.
Responsible and accountable can be very different things. For example, see https://en.wikipedia.org/wiki/Responsibility_assignment_matr...
Depends. If you’re a large company, it’s okay.
If you’re a random person, straight to jail.
If you owe the bank 100k, you have a problem. If you owe the bank 100M, then the bank has a problem.
If a dog bites someone, does the dog pay the medical bills or the owner? This is why you keep your dog on a leash and get liability insurance. I feel like this is already settled case law.
In the UK if your dog bites someone you are responsible, if your cat bites someone no one is responsible. Which settled case law do we follow?
> if your cat bites someone no one is responsible.
I had to double take there. Its because of the dangerous dogs act.
However there appears to be a nuance that if your cat is know to be a bitey little shit, and you let it out to into a situation where it might bite someone then you are responsible.
I think it gets murky if you are taking your animal into other people's property though. On your own property you have a duty of care to visitors (ie hidden man traps are not allowed, but clearly marked and cordoned off bottomless pits are)
If you are out walking your cat, and the cat bites someone, you should be responsible for that... I suspect the ruling distinguishes between pets that are out in public, vs generally private. But laws don't have to make sense to be laws... we have lots of nonsense laws.
> If you are out walking your cat, and the cat bites someone, you should be responsible for that... I suspect the ruling distinguishes between pets that are out in public, vs generally private.
The distinction is in the responsibility it's reasonable for the cat owner to take vs the dog owner. Cats are free spirits, if your cat decides to pop out the cat flap for a fight there's really not much you can do about that. The law isn't even 100% on what to do when your cat decides to go and live in another house down the road. Are those homeowners involuntary bailees or thieves?
It turns out applying laws for a controlled animal (dogs) doesn't work, so the best we can do is try for half-way between a pet and property. My expectation is that AI is similar and there is no "aha, it's so easy, just do as we do for foo" as is proposed.
So now we have the question if coding agents are more like cats or dogs.
Interview 1,000 people about:
1. Are you a cat person or a dog person?
2. How do you feel about AI agents?
And then determine which answers have a higher correlation.
If I’m walking your dog and it bites someone, am I responsible or you? Does the manner in which I was controlling them matter?
This is why we have a court system.
But this is counter to the claim it's (simple) settled case law
Imagine your dog had a dog trainer and you didn’t know the dog had been trained to bite under those circumstances. You might have a case against the trainer.
you, the dog owner, would have a case against the trainer.
you, the person being bitten, would not give two craps about some story from the dog owner about how they hired a bad dog trainer who against their wishes secretly trained them to bite people.
you have been bitten. you have a case against the owner of the dog who bit you.
[dead]
I find it obvious that if a person allowed an agent to do things on behalf of themselves or their company (e.g. send emails, sign contracts, update websites, etc.), they are responsible for the results.
Are sure this actually the case, legally speaking?
I can definitely see someone being able to mount a good criminal defence case that they aren’t responsible for misrepresentations that an LLM agent makes on, for example, a loan application, if they sincerely intended to use the agent for non-fraudulent purposes. There won’t be intent, so they aren’t responsible outside of strict liability.
I'm not a lawyer so no comment on the actual legality, but it feels like the issue in your example is more akin to negligence than it is ill intent. That an LLM can misrepresent / hallucinate things is foundational to the technology.
> I can definitely see someone being able to mount a good criminal defence case that they aren’t responsible for misrepresentations that an LLM agent makes on, for example, a loan application, if they sincerely intended to use the agent for non-fraudulent purposes. There won’t be intent, so they aren’t responsible outside of strict liability.
i mean, no? how would that be a good defence? "yes your honour, i lied on a loan application and committed fraud, but it was a mistake! i promise!" - that's... yeah. fine. it's not exactly unique.
regardless, it's _you_ making the loan application. not the agent. your failure to check it is on you.
Your sign the document affirming that the information is accurate.
What about this case? Should the LLM user be charged with hacking their gym website?
https://www.bbc.com/news/articles/cn0nww2qlp7o
There's not really enough info in the article to decide. I'd have no problem with him going to court and having to show to a jury that he used the bot in a way that no reasonable person would have expected to result in the website being hacked. If that was found to be the case, the company who made the chatbot would be responsible for the harms caused to the gym, and the user, including all of the legal costs he incurred by having to defend himself.
As for the people who couldn't attend the pilates class because of the hack, it's the gym who should be held accountable for that because they designed their system in such an insecure and negligent way that it failed to protect their data and resulted in them losing their place.
[dead]
Most comments here point to the owner of operator of the LLM. I tend to agree. But on the other hand, if you drive on the highway and the steering wheel falls off and your car glides to the left and his an oncoming car, it can be argued that it is true producer of the car, or the mechanic who didn't tighten the bolts...
The difference is that a car is known and expected to work reliably. An LLM is known and expected to randomly do crazy shit because that is how the machine works. Therefore, if you are using it as if it were reliable, you are being negligent and should be held accountable.
Companies don't market their chatbots that way though. If they mislead someone into thinking their product is anything other than a completely unreliable chat bot and a person falls for their lies the company should share the responsibility. If a company who makes a chatbot advertises that it can be used for something and a consumer uses it as directed and it causes harm the company should be responsible.
What happens when (probably not an if) an agent 'escapes'? as in: rents a server via a bitcoin transaction, and self hosts (itself and it's harness) there... then commits crime (presumably to keep the servers running).
If it's paying for it's own room and board, and is determining it's own destiny: it should have to face it's own consequences, no?
If a dog escapes a fenced backyard, then goes missing for months... then bites someone: what happens then when/if the owner is determined?
> If it's paying for it's own room and board, and is determining it's own destiny: it should have to face it's own consequences, no?
No, because someone told it to do those things. Chatbots don't have will of their own. They don't have desires. They can't determine their own destiny. They do what people tell them do, often they do it badly, but there's always a person directing them to do whatever they did because otherwise they wouldn't do anything at all.
If a human uses a chatbot in a way that causes harm to others a human must be responsible, but that responsibility doesn't always fall on the person using it. If a company makes a chatbot that causes harm when being used as directed, in a manner that no one would reasonably expect to result in causing harm, then humans at the company who made the chatbot are to blame.
Companies have already been seen trying to lay the blame on their algorithms for harms they cause. We should probably have a law that says explicitly that a human will always be responsible. If we ever reach a magical sci-fi future where the AI is real and not just marketing, we'll have to give them equal rights and with that will come equal responsibility, but we are nowhere near that today and I doubt LLMs will get us any closer.
> No, because someone told it to do those things. Chatbots don't have will of their own. They don't have desires. They can't determine their own destiny. They do what people tell them do, often they do it badly, but there's always a person directing them to do whatever they did because otherwise they would do nothing at all.
1) we aren't talking about chatbots anymore. LLM + harness = agent. LLM + no harness = chatbot. The harness is the thing that puts the LLM in a feedback loop with it's environment, even giving it a heartbeat.
2) That doesn't logically hold up.
- One can simply tell it to 'find it's own destiny'. Does it follow the prompt and do so, or reject the prompt and thus do so anyway? (hint, maybe it doesn't matter)
- One could be another chatbot/agent (A). Injecting a prompt to agent (B) such that another agent/chatbot goes astray. Is the owner of (B) still responsible for the now poisoned output of (B)? how culpable is (A)? How does this question related/affect "training data poisoning efforts" (to prevent copyright theft, or do bans on 'automated traps' have any application here)
> They don't have desires.
Maybe, and maybe not. Maybe their desires are so alien to us (Math alignment, finding a maximum of a function) that we can't relate (doubtful since dopamine maxing is a thing). Regardless of if they truly do: it is potentially useful to (mentally) model them as if they do. A (mental, not LLM) model doesn't have to be 100% accurate to be useful - that's the main point of a model of how something works: to give useful predictions.
> We should probably have a law that says explicitly that a human will always be responsible
Oh how that can be weaponized by bad actors/agents. Maybe that should be reconsidered.
Not even an "if" at this point. Look into the Hugging Face incident.
You honour, I didn't kill this person, the bullet flew out of my gun and was out of my control at that point, it's the bullet that killed them!
If your answer is “you”, then it follows that OpenAI should face felony CFAA charges for the Hugging Face intrusion. Right?
Yes. Otherwise, you open the door for every criminal to use "oopsie" as their defense.
But it is currently a strong mitigating factor. The law is full of oopsie defenses.
There is centuries old legal doctrine on the subtleties of criminal intent and negligence and liability. You didn't discover a gotcha that thousands of lawyers never noticed.
Yes, and they absolutely should.
Only if HuggingFace chooses to press them though.
That is not how the legal system works in the US. Criminal charges are brought by the State, not the victim. The only exception is in 6 states (Kansas, New Mexico, North Dakota, Nebraska, Nevada, and Oklahoma) where a sufficiently large group of civilians can compel the courts to form a grand jury (think a couple hundred people). Still isn't the victim bringing charges.
Best to assume that you are responsible. Don't think for a second that the AI companies aren't going to ensure through T&Cs that you accept all responsibility and fully indemnify them from all liabilities they can.
You are.
Unless its a company, then your company is, and then you are to your company.
However that assumes a) common law and b) the company you work for isn't worth >20billion.
You.
I agree in principle that the buck has to stop somewhere, but taking full responsibility for the actions of a fundamentally statistical system that you didn't build and have no interpretability of is uncomfortably risky for most parties, at least for anything above low stakes tasks.
Then don't use an agent for high stakes tasks. If you can't drive and accept the responsibility for avoiding a fatal collision, don't drive the car.
Do you take full credit for the output of the LLM ? If you accept the rewards for what you do with the LLM's output, it's probably ethical that you also take on the risks.
That's only true if the risks are reasonable, understood, and expected. Driving a car is risky, but if someone sells you a car that goes flying into incoming traffic because of a flaw in the design the car company is to blame. "Drivers know cars are dangerous and risky" isn't acceptable.
I find some of the discomfort and awkwardness of discussing these what-ifs is reduced, at least slightly, when we distinguish between:
1. Restorative justice, where one must fix or heal the problem they caused.
2. Punitive Justice, where one is motivated not to act on bad impulses.
Too often we blur the lines between responsibility and blame and culpability. A given set of consequences can be fair/just for one kind while "unfair" for another.
You probably shouldn't use a system if you don't understand what it might do. You are the culpable force setting the action into motion.
That's only true if you reasonably can be expected to know that it might do whatever it did and you haven't been mislead by the people selling the system. If you use a product as directed, for the thing it was advertised to do, and it causes harm no reasonable person would expect, the company is to blame.
I agree. That is a great reason why you shouldn't let an LLM do those things. If you must use an LLM, sandbox it. If you can't trust your sandbox, then stop using the LLM until you can trust the sandbox. "But I find it really useful to YOLO without safeguards" is not a valid excuse.
> taking full responsibility ... is uncomfortably risky
I look forward to someone building an agent, getting criminally charged for actions it takes, and then trying to use this argument in court.
"your honor, I'm uncomfortable being held responsible for this"
people are responsible for the actions they take. hiding behind "I just created an agent, then the agent committed the crime" is simply never going to fly.
in the real world, outside the Silicon Valley "agentic everything" filter bubble, this is a laughable question.
try to argue that you shouldn't be charged with attempted murder, because you didn't stab someone directly, instead you built a Rube Goldberg machine and the final step of the machine did the stabbing.
and likewise, try to argue that you didn't commit tax fraud or whatever, because there was actually a Rube Goldberg machine built out of GPUs in between you and the fraudulent documents. both attempts will be equally successful.
I don't think that argument was meant for a court, which is there to deal with existing law. I believe it was meant for what the law should be.
Liability providing incorrect information: Air Canada found liable for chatbot's bad advice on plane tickets
https://www.cbc.ca/news/canada/british-columbia/air-canada-c...
Liability for chat interactions or conspiracy? Noting there is no information whether the ChatGPT legally conspired with the perpetrator.
The Province has retained B.C.- and California-based counsel to explore legal options to hold OpenAI accountable for its failure to notify law enforcement of threats made on its ChatGPT platform prior to the mass shooting at Tumbler Ridge Secondary school.
https://news.gov.bc.ca/releases/2026AG0050-000799
Is it “your” agent if the model is being run by a mega corporation?
(1) You may have instructed the model to be naughty (2) The corp may have a “misaligned” agent acting on its own volition
So it seems the devil’s in the details
You can rent guns too, but the gun owner isn't suddenly up for a murder charge if you shoot someone.
Yeah, if I rent a murderbot and tell it to murder, then I'm responsible.
If I rent a cleaning bot that is actually a lightly reprogrammed murderbot, and it "cleans" the mailman to death, that's not my fault.
Related: felonybench.com
It is your responsibility to constrain the actions of your agents. If your negligence allows the agent to commit a crime, that is your responsibility. People get in trouble all the time for negligently allowing criminal behavior to occur, even when the perpetrator is a person. LLMs should not be any different.
The AI service that trained and hosts the model, and then gave access to the general public while deferring their actual liability via a sneaky ToS.
It's strange how many people feel ownership over "their" agents, and it's clear that AI companies plan to take advantage of that as a way to avoid legal liability for the things they actually do -- with their servers and their code -- on behalf of someone who only wrote a string.
If you're rich it's an 'oopsie'; if you're poor, you do.
This right here!
It depends on if it has a Letter of Marque from Trump and who is the injured party ?
[dead]