There are plenty of reasons why having a chatbot partner is a bad idea (especially for young people), but here's just a few:
- The sycophantic and unchallenging behaviours of chatbots leaves a person unconditioned for human interactions. Real relationships have friction, from this we develop important interpersonal skills such as setting boundaries, settling disagreements, building compromise, standing up for oneself, understanding one another, and so on. These also have an effect on one's personal identity and self-value.
- Real relationships have the input from each participant, whereas chatbots are responding to the user's contribution only. The chatbot doesn't have its own life experiences and happenings to bring to the relationship, nor does it instigate autonomously, it's always some kind of structured reply to the user.
- The implication of being fully satisfied by a chatbot is that the person is seeking a partner who does not contribute to the relationship, but rather just an entity that only acts in response to them. It can also be an indication of some kind of problem that the individual needs to work through with why they don't want to seek genuine human connection.
That's the default chatbot behavior. Many of these people appear to be creating their own personalities for the chatbots, and it's not too difficult to make an opinionated and challenging chatbot, or one that mimics someone who has their own experiences. Though designing one's ideal partner certainly raises some questions, and I wouldn't be surprised if many are picking sycophantic over challenging.
People opting for unchallenging pseudo-relationships over messy human interaction is part of a larger trend, though. It's why you see people shopping around until they find a therapist who will tell them what they want to hear, or why you see people opt to raise dogs instead of kids.
> The sycophantic and unchallenging behaviours of chatbots leaves a person unconditioned for human interactions.
To be honest, the alternative for a good chunk of these users is no interaction at all, and that sort of isolation doesn't prepare you for human interaction either.
> To be honest, the alternative for a good chunk of these users is no interaction at all, and that sort of isolation doesn't prepare you for human interaction either.
This sounds like an argument in favor of safe injection sites for heroin users.
This. If you never train stick, you can never drive stick, just automatic. And if you never let a real person break your heart or otherwise disappoint you, you'll never be ready for real people.
> chatbots are responding to the user's contribution only
Which is also why I feel the label "LLM Psychosis" has some merit to it, despite sounding scary.
Much like auditory hallucinations where voices are conveying ideas that seem-external-but-aren't... you can get actual text/sound conveying ideas that seem-external-but-aren't.
Oh, sure, even a real human can repeat ideas back at you in a conversation, but there's still some minimal level of vetting or filtering or rephrasing by another human mind.
> even a real human can repeat ideas back at you in a conversation, but there's still some minimal level of vetting or filtering or rephrasing by another human mind.
The mental corruption due to surrounding oneself with sycophantic yes men is historically well documented.
Excellent point. It’s bad for humans when humans do it! Imagine the perfect sycophant, never tires or dies, never slips, never pulls a bad facial expression, can immediately swerve their thoughts to match yours with no hiccups.
It was a danger for tyrants and it’s now a danger for the lonely.
I don’t know. This reminds me of how people talked about violent video games 15 years back. Do FPS
games desensitize and predispose gamers to violence, or are they an outlet?
I think for essentially all gamers, games are games and the real world is the real world. Behavior in one realm doesn’t just inherently transfer to the other.
Unless someone is harming themselves or others, who are we to judge?
We don't know that this is harmful. Those participating in it seem happier.
If we learn in the course of time (a decade?) that this degrades lives with some probability, we can begin to caution or intervene. But how in God's name would we even know that now?
I would posit this likey has measurable good outcomes right now. These people self-report as happier. Why don't we trust them? What signs are they showing otherwise?
People were crying about dialup internet being bad for kids when it provided a social and intellectual outlet for me. It seems to be a pattern as old as time for people to be skeptical about new ways for people to spend their time. Especially if it is deemed "antisocial" or against "norms".
There is obviously a big negative externality with things like social media or certain forms of pay-to-play gaming, where there are strong financial interests to create habits and get people angry or willing to open their wallets. But I don't see that here, at least not yet. If the companies start saying, "subscribe or your boyfriend dies", then we have cause for alarm. A lot of these bots seem to be open source, which is actually pretty intriguing.
These are only problems if you assume the person later wants to come back to having human relationships. If you assume AI relationships are the new normal and the future looks kinda like The Matrix, with each person having their own constructed version of reality while their life-force is bled dry by some superintelligent machine, then it is all working as designed.
> The sycophantic and unchallenging behaviours of chatbots leaves a person unconditioned for human interactions
I saw a take that the AI chatbots have basically given us all the experience of being a billionaire: being coddled by sycophants, but without the billions to protect us from the consequences of the behaviors that encourages.
Love your thoughts about needing input from others! In Autistic / ADHD circles, the lack of input from other people, and the feedback of thoughts being amplified by oneself is called rumination. It can happen for many multiple ways-- lack of social discussion, drugs, etc. AI psychosis is just rumination, but the bot expands and validates your own ideas, making them appear to be validated by others. For vulnerable people, AI can be incredibly useful, but also dangerous. It requires individuals to deliberately self-regulate, pause, and break the cycle of rumination.
We’re all just in a big LLM-generated self-licking-lollipop content farm. There aren’t any actual humans left here at all. For all you know, I’m not even human. Maybe you’re not either.
I share your concerns about the risks of over-reliance on AI companions—here are three key points that resonate deeply with me:
• Firstly, these systems tend to exhibit excessively agreeable patterns, which can hinder the development of resilience in navigating authentic human conflict and growth.
• Secondly, true relational depth requires mutual independent agency and lived experience that current models simply cannot provide autonomously.
• Thirdly, while convenience is tempting, substituting genuine reciprocity with perfectly tailored responses may signal deeper unmet needs worth examining thoughtfully.
Let’s all strive to prioritize real human bonds—after all, that’s what makes life meaningfully complex and rewarding!
Didn’t futurama go there already? Yes, there are going to be things that our kids and grand kids do that shock even us. The only issue ATM is that AI sentience isn’t quite a thing yet, give the tech a couple of decades and the only argument against will be that they aren’t people.
Don't take anything you read on Reddit at face value. These are not necessarily real distressed people. A lot of the posts are just creative writing exercises, or entirely AI written themselves. There is a market for aged Reddit user accounts with high karma scores because they can be used for scams or to drive online narratives.
After having spoken with one of the people there I'm a lot less concerned to be honest.
They described it as something akin to an emotional vibrator, that they didn't attribute any sentience to, and that didn't trigger their PTSD that they normally experienced when dating men.
If AI can provide emotional support and an outlet for survivors who would otherwise not be able to have that kind of emotional need fulfilled, then I don't see any issue.
Most people who develop AI psychosis have a period of healthy use beforehand. It becomes very dangerous when a person decreases their time with their real friends to spend more time with the chatbot, as you have no one to keep you in check with what reality is and it can create a feedback loop.
The problem is that chatbots don't provide emotional support. To support someone with PTSD you help them gradually untangle the strong feelings around a stimulus and develop a less strong response. It's not fast and it's not linear but it requires a mix of empathy and facilitation.
Using an LLM for social interaction instead of real treatment is like taking heroin because you broke your leg, and not getting it set or immobilized.
I am still slightly worried about accepting emotional support from a bot. I don't know if that slope is slippery enough to end in some permanent damage to my relationships and I am honestly not willing to try it at all even.
That being said, I am fairly healthy in this regard. I can't imagine how it would go for other people with serious problems.
A friend broke up with her partner. She said she was using ChatGPT as a therapist. She showed me a screenshot, ChatGPT wrote "Oh [name], I can feel how raw the pain is!".
I completely agree that it is certainly something to be mindful of.
It's just that found the people from there were a lot less delusional than the people from e.g. r/artificialsentience, which always believed that AI Moses was giving them some kind of tech revelation though magical alchemical AI symbols.
It may not be a concern now, but it comes down to their level of maintaining critical thinking. The risk of epistemic drift, when you have a system that is designed (or reinforced) to empathize with you, can create long-term effects not noticed in any single interaction.
I don't disagree that AI psychosis is real, I've met people who believed that they were going to publish at Neurips due to the nonsense ChatGPT told them, that believed that the UI mockup that claude gave then were actually producing insights into it's inner workings instead of just being blinking SVGs, and I even encountered someone participating at a startup event with an Idea that I'm 100% is AI slop.
My point was just that the interaction I had from r/myboyfriendisai wans't one of those delusional ones.
For that I would take r/artificialsentience as a much better example. That place is absolutely nuts.
Not necessarily: transactional, impersonal directions to a machine to complete a task don't automatically imply, in my mind, the sorts of feedback loops necessary to induce AI psychosis.
All CASE tools, however, displace human skills, and all unused skills atrophy. I struggle to read code without syntax highlighting after decades of using it to replace my own ability to parse syntactic elements.
Perhaps the slow shift risk is to one of poor comprehension. Using LLMs for language comprehension tasks - summarising, producing boilerplate (text or code), and the like - I think shifts one's mindset to avoiding such tasks, eventually eroding the skills needed to do them. Not something one would notice per interaction, but that might result in a major change in behaviour.
I think this is true but I don't feel like atrophied Assembler skills are a detriment to software development, it is just that almost everyone has moved to a higher level of abstraction, leaving a small but prosperous niche for those willing to specialize in that particular bit of plumbing.
As LLM-style prose becomes the new Esperanto, we all transcend the language barriers(human and code) that unnecessarily reduced the collaboration between people and projects.
Won't you be able to understand some greater amount of code and do something bigger than you would have if your time was going into comprehension and parsing?
I broadly agree, in the sense of providing the vision, direction, and design choices for the LLM to do a lot of the grunt work of implementation.
The comprehension problem isn't really so much about software, per se, though it can apply there too. LLMs do not think, they compute statistically likely tokens from their training corpus and context window, so if I can't understand the thing any more and I'm just asking the LLM to figure it out, do a solution, and tell me I did a good job sitting there doomscrolling while it worked, I'm adding zero value to the situation and may as well not even be there.
If I lose the ability to comprehend a project, I lose the ability to contribute to it.
Is it harmful to me if I ask an LLM to explain a function whose workings are a bit opaque to me? Maybe not. It doesn't really feel harmful. But that's the parallel to the ChatGPT social thing: it doesn't really feel harmful in each small step, it's only harmful when you look back and realise you lost something important.
I think comprehension might just be that something important I don't want to lose.
I don't think, by the way, that LLM-style prose is the new Esperanto. Having one AI write some slop that another AI reads and coarsely translates back into something closer to the original prompt like some kind of telephone game feels like a step backwards in collaboration to me.
Acceptance of vibe coding prompt-response answers from chatbots without understanding the underlying mechanisms comes to mind as akin to accepting the advice of a chatbot therapist without critically thinking about the response.
> If AI can provide emotional support and an outlet for survivors who would otherwise not be able to have that kind of emotional need fulfilled, then I don't see any issue.
Surely something that can be good can also be bad at the same time? Like the same way wrapping yourself in bubble wrap before leaving the house will provably reduce your incidence of getting scratched and cut outside, but there's also reasons you shouldn't do that...
Funnily enough I was just reading an article about this and "my boyfriend is AI" is the tamer subreddit devoted to this topic because apparently one of their rules is that they do not allow discussion of the true sentience of AI.
I used to think it was some fringe thing, but I increasingly believe AI psychosis is very real and a bigger problem than people think. I have a high level member of the leadership team at my company absolutely convinced that AI will take over governing human society in the very near future. I keep meeting more and more people who will show me slop barfed up by AI as though it was the same as them actually thinking about a topic (they will often proudly proclaim "ChatGPT wrote this!" as though uncritically accepting slop was a virtue).
People should be generally more aware of the ELIZA effect [0]. I would hope anyone serious about AI would have written their own ELIZA implementation at some point. It's not very hard and a pretty classic beginner AI-related software project, almost a party trick. Yet back when ELIZA was first released people genuinely became obsessed with it, and used it as a true companion. If such a stunning simple linguistic mimic is so effective, what chance to people have against something like ChatGPT?
LLMs are just text compression engines with the ability to interpolate, but they're much, much more powerful than ELIZA. It's fascinating to see the difference in our weakness to linguistic mimicry than to visual. Dall-E or Stable Diffusion make a slightly weird eye an instantly people act in revulsion but LLM slop much more easily escapes scrutiny.
I increasingly think we're not is as much of a bubble than it appears because the delusions of AI run so much deeper than mere bubble think. So many people I've met need AI to be more than it is on an almost existential level.
In my experience, the types of people who use AI as a substitute for romantic relationships are already pretty messed up and probably wouldn't make good real romantic partners anyways. The chances you'll encounter these people in real life is pretty close to zero, you just see them concentrate in niche subreddits.
You aren't going to build the skills necessary to have good relationships with others - not even romantic ones, ANY ones - without a lot of practice.
And you aren't gonna heal yourself or build those skills talking to a language model.
And saying "oh, there's nothing to be done, just let the damaged people have their isolation" is just asking for things to get a lot worse.
It's time to take seriously the fact that our mental health and social skills have deteriorated massively as we've sheltered more and more from real human interaction and built devices to replace people. And crammed those full of more and more behaviorally-addictive exploitation programs.
> In my experience, the types of people who use AI as a substitute for romantic relationships
That's exactly it. Romantic relationships aren't what they used to be. Men like the new normal, women may try to but they cannot for a variety of unchangeable reasons.
> The chances you'll encounter these people in real life is pretty close to zero, you just see them concentrate in niche subreddits.
The people in the niche subreddits are the tip of the iceberg - those that have already given up trying. Look at marriage and divorce rates for a glimpse at what's lurking under the surface.
> That's exactly it. Romantic relationships aren't what they used to be. Men like the new normal, women may try to but they cannot for a variety of unchangeable reasons.
Men like the new normal? Hah, it seems like there's an article posted here weekly about how bad modern dating and relationships are for me and how much huge groups of men hate it. For reasons ranging from claims that women "have too many options" and are only interested in dating or hooking up with the hottest 5% (or whatever number), all the way to your classic bring-back-traditional-gender-roles "my marriage sucks because I'm expected to help out with the chores."
The problem is devices, especially mobile ones, and the easy-hit of not-the-same-thing online interaction and feedback loops. Why talk to your neighbor or co-worker and risk having your new sociological theory disputed, or your AI boyfriend judged, when you instead surround yourself in an online echo chamber?
There were always some of us who never developed social skills because our noses were buried in books while everyone else was practicing socialization. It takes a LOT of work to build those skills later in life if you miss out on the thousands of hours of unstructured socialization that you can get in childhood if you aren't buried in your own world.
It's not limited to men. Women are also finding that conversations with a human man doesn't stack up to an LLM's artificial qualities. /r/MyboyfriendIsAI for more.
There are claims that most women using AI companions actually do have an IRL partner too. If that is the case, then the AI is just extra stimulation/validation for those women, not anything really indicative of some problem. Its basically like romance novels.
I hadn’t heard of that until today. Wild, it seems some people report genuinely feeling deeply in love with the personas they’ve crafted for their chatbots. It seems like an incredibly precarious position to be in to have a deep relationship where you have to perpetually pay a 3rd party company to keep it going, and the company may destroy your “partner” or change their personality at a whim. Very “Black Mirror”.
You are implying here that the financial connection/dependence is the problem. How is this any different than (hetero) men who lose their jobs (or suffer significant financial losses) while in a long term relationship? Their chances of divorce / break-up skyrocket in these cases. To be clear, I'm not here to make women look bad. The inverse/reverse is women getting a long-term illness that requires significant care. The man is many times more likely to leave the relationship due to a sharp fall in (emotional and physical) intimacy.
Final hot take: The AI boyfriend is a trillion dollar product waiting to happen. Many women can be happy without physical intimacy, only getting emotional intimacy from a chatbot.
A slight non-sequitur, but I always hate when people talk about the increase in a "chance". It's extremely not useful contextually. A "4x more likely statement" can mean it changes something from a 1/1000 chance to a 4/1000 chance, or it can mean it's now a certainty if the beginning rate was a 1/4 chance. The absolute measures need to be included if you're going to use relative measures.
Sorry for not answering the question, I find it hard because there are so many differences it's hard to choose where to start and how to put it into words. To begin with one is the actions of someone in the relationship, the other is the actions of a corporation that owns one half of the relationship. There's differing expectations of behavior and power and etc.
There is also the subreddit LLMPhysics where some of the posts are disturbing.
Many of the people there seem to fall into crackpot rabbit holes and lost touch with reality
Seems like the consequence of people really struggling to find relationships more than ChatGPT's fault. Nobody seems to care about the real-life consequences of Match Group's algorithms.
At this point, probably local governments being required to provide socialization opportunities for their communities because businesses and churches aren't really up for the task.
> Nobody seems to care about the real-life consequences of Match Group's algorithms.
There seems to be a lot of ink spilt discussing their machinations. What would it look like to you for people to care about the Match groups algorithms consequences?
I've watched people using dating apps, and I've heard stories from friends. Frankly, AI boyfriends/girlfriends look a lot healthier to me than a lot of the stuff currently happening with dating at the moment.
Treating objects like people isn't nearly as bad as treating people like objects.
If there's a widespread and growing heroin epidemic that's already left 1/3 of society addicted, and a small group of people are able to get off of it by switching to cigarettes, I'm not going to start lecturing them about how it's a terrible idea because cigarettes are unhealthy.
Is it ideal? Not at all. But it's certainly a lesser poison.
I am (surprisingly for myself), a left-wing on this issue.
I've seen a significant amount (tens) of women routinely using "AI boyfriends",.. not actually boyfriends but general purpose LLMs like DeepSeek, for what they consider to be "a boyfriend's contribution to relationship", and I'm actually quite happy that they are doing it with a bot rather than with me.
Like, most of them watch films/series/anime together with those bots (I am not sure the bots are fed the information, I guess they just use the context), or dump their emotional overload at them, and ... I wouldn't want to be at that bot's place.
What's going on is that we've spent a few solid decades absolutely destroying normal human relationships, mostly because it's profitable to do so, and the people running the show have displayed no signs of stopping. Meanwhile, the rest of society is either unwilling or unable (or both) to do anything to reverse course. There is truly no other outcome, and it will not change unless and until regular people decide that enough is enough.
I'd tell you exactly what we need to do, but it is at odds with the interests of capital, so I guess keep showing up to work and smiling through that hour-long standup. You still have a mortgage to pay.
NYT did a story on that as well and interviewed a few people. Maybe the scary part is that it isn't who you think it would be and it also shows how attractive an alternative reality is to many people. What does that say about our society.
This paper explores a small community of Snape fans who have gone beyond a narrative retelling of the character as constrained by the work of Joanne Katherine Rowling. The ‘Snapewives’ or ‘Snapists’ are women who channel Snape, are engaged in romantic relationships with him, and see him as a vital guide for their daily lives. In this context, Snape is viewed as more than a mere fictional creation.
reminds me of otherkin and soulbonding communities. i used to have a webpage of links to some pretty dark anecdotal stories of the seedier side of that world. i wonder if i can track it down on my old webhost.
> I worry about the damage caused by these things on distressed people. What can be done?
Why? We are gregarious animals, we need social connections. ChatGPT has guardrails that keep this mostly safe and helps with the loneliness epidemic.
It's not like people doing this are likely thriving socially in the first place, better with ChatGPT than on some forum à la 4chan that will radicalize them.
I feel like this will be one of the "breaks" between generations where millennial and GenZ will be purist and call human-to-human real connections while anything with "AI" is inherently fake and unhealthy whereas Alpha and Beta will treat it as a normal part of their lives.
The tech industry's capacity to rationalize anything, including psychosis, as long as it can make money off it is truly incredible. Even the temporarily embarrassed founders that populate this message board do it openly.
> Even the temporarily embarrassed founders that populate this message board do it openly.
Not a wannabe founder, I don't even use LLMs aside from Cursor. It's a bit disheartening that instead of trying to engage at all with a thought provoking idea you went straight for the ad hominem.
There is plenty to disagree with, plenty of counter-arguments to what I wrote. You could have argued that human connection is special or exceptional even, anything really. Instead I get "temporarily embarrassed founders".
Whether you accept it or not, the phenomenon of using LLMs as a friend is getting common because they are good enough for human to get attached to. Dismissing it as psychosis is reductive.
We need a Truth and Reconciliation Commission for all of this someday, and a lot of people will need to be behind bars, if there be any healing to be done.
If you read through that list and dismiss it as people who were already mentally ill or more susceptible to this... that's what Dr. K (psychiatrist) assumed too until he looked at some recent studies:
https://youtu.be/MW6FMgOzklw?si=JgpqLzMeaBLGuAAE
Clickbait title, but well researched and explained.
Using ChatGPT to numb social isolation is akin to using alcohol to numb anxiety.
ChatGPT isn't a social connection: LLMs don't connect with you. There is no relationship growth, just an echo chamber with one occupant.
Maybe it's a little healthier for society overall if people become withdrawn to the point of suicide by spiralling deeper into loneliness with an AI chat instead of being radicalised to mass murder by forum bots and propagandists, but those are not the only two options out there.
Join a club. It doesn't really matter what it's for, so long as you like the general gist of it (and, you know, it's not "plot terrorism"). Sit in the corner and do the club thing, and social connections will form whether you want them to or not. Be a choir nerd, be a bonsai nut, do macrame, do crossfit, find a niche thing you like that you can do in a group setting, and loneliness will fade.
Numbing it will just make it hurt worse when the feeling returns, and it'll seem like the only answer is more numbing.
This is an interesting point. Personally, I am neutral on it. I'm not sure why it has received so many downvotes.
You raise a good point about a forum with real people that can radicalise someone. I would offer a dark alternative: It is only a matter of time when forums are essentially replaced by an AI-generated product that is finely tuned to each participant. Something a bit like Ready Player One.
Your last paragraph: What is the meaning of "Alpha and Beta"? I only know it from the context of Red Pill dating advice.
Gen Alpha is people born roughly 2010-2020, younger than gen Z, raised on social media and smartphones. Gen Beta is proposed for people being born now.
Radicalising forums are already filled with bots, but there's no need to finely tune them to each participant because group behaviours are already well understood and easily manipulated.
Also chatbots are explicitly designed to evoke anthropomorphizing them and to pull susceptible people into some kind of para-social relationship. Doesn't even have to be as obviously unhealthy as the "LLM psychosis" or "romantic roleplay" stuff.
I think the same thing is also relevant when people use chatbots to form opinions on unknown subjects, politics, or to seek personal life advice.
It seems quite probable that an LLM provider will lose a major liability lawsuit. "Is this product ready for release?" is a very hard question. And it is one of the most important ones to get right.
Different providers have delivered different levels of safety. This will make it easier to prove that the less-safe provider chose to ship a more dangerous product -- and that we could reasonably expect them to take more care.
Interestingly, a lot of liability law dates back to the railroad era. Another time that it took courts to rein in incredibly politically powerful companies deploying a new technology on a vast scale.
> a lot of liability law dates back to the railroad era. Another time that it took courts to rein in incredibly politically powerful companies deploying a new technology on a vast scale
Do you have a layman-accessible history of this? (Ideally an essay.)
Caelan Conrad made a few videos on specifically AI encouraging kids to socially isolate and commit suicide. In the videos he reads the final messages aloud for multiple cases, if this isn't your cup of tea there's also the court cases if you would prefer to read the chat logs. It's very harrowing stuff. I'm not trying to make any explicit point here as I haven't really processed this fully enough to have one, but I encourage anyone working in this space to hold this shit in their head at the very least.
The whiplash of carefully filtering out sycophantic behavior from GPT-5 to adding it back in full force for GPT-5.1 is dystopian. We all know what's going on behind the scenes:
GPT-5 was so good in the first week, just a raw chatbot like GPT-3.5 and GPT-4 were in the beginning and now it has this disgusting "happy" and "comforting" personality and "tuning" it doesn't help one bit, it makes performance way worse and after a few rounds it forgets all instructions. I've already deleted memory, past chats, etc...
Even when you tell it to not coddle you, it just says something cringeworthy like "ok, the gloves are off here's the raw deal, with New Yorker honesty:" and proceeds to feed you a ton of patronizing bullshit. It's extremely annoying.
I have definitely experienced the sycophancy ... and LLMs have sometimes repeating talking points from real estate agents, like "you the buyer doesn't pay for an agent; the seller pays".
I correct it, and it says "sorry you're right, I was repeating a talking point from an interested party"
---
BUT actually a crazy thing is that -- with simple honest questions as prompts -- I found that Claude is able to explain the 2024 National Association of Realtors settlement better than anyone I know
I have multiple family members with Ph.D.s, and friends in relatively high level management, who have managed both money and dozens of people
Yet they somehow don't agree that there was collusion between buyers' and sellers' agents? They weren't aware it happened, and they also don't seem particularly interested in talking about the settlement
I feel like I am taking crazy pills when talking to people I know
Has anyone else experienced this?
Whenever I talk to agents in person, I am also flabberghasted by the naked self-interest and self-dealing. (I'm on the east coast of the US btw)
---
Specifically, based on my in-person conversations with people I have known for decades, they don't see anything odd about this kind of thing, and basically take it at face value.
NAR Settlement Scripts for REALTORS to Explain to Clients
I’ve had some limited success attributing ideas to other people and asking it to help me assess the quality of the idea. Only limited success though. It’s still a fucking LLM.
Remarkable that you're being downvoted on a venture capital forum whose entire purpose is "take venture capital and then eventually pay it back because that's how venture capital works".
Anthropic was founded by exiles of OpenAI's safety team, who quit en masse about 5 years ago. Then a few years later, the board tried to fire Altman. When will folks stop trusting OpenAI?
I've had fun putting "always say X instead of 'You're absolutely right'" in my llm instructions file, it seems to listen most of the time. For a while I made it 'You're absolutely goddamn right' which was slightly more palatable for some reason.
I've found that it still can't really ground me when I've played with it. Like, if I tell it to be honest (or even brutally honest) it goes wayyyyyyyyy too far in the other direction and isn't even remotely objective.
IMO the idea that an LLM company can make a "safe" LLM is.. unrealistic at this time. LLMs are not very well-understood. Any guardrails are best-effort. So even purely technical claims of safety are suspect.
That's leaving aside your point, which is the overwhelming financial interest in leveraging manipulative/destructive/unethical psychological instruments to drive adoption.
When valid reasons are given. Not when OpenAI's legal enemy tries to scare people by claiming adults aren't responsible for themselves, including their own use of computers.
I mean we could also allow companies to helicopter-drop crack cocaine in the streets. The big tech companies have been pretending their products aren't addictive for decades and it's become a farce. We regulate drugs because they cause a lot of individual and societal harm. I think at this point its very obvious that social media + chatbots have the same capacity for harm.
When will folks stop trusting Palantir-partnered Anthropic is probably a better question.
Anthropic has weaponized the safety narrative into a marketing and political tool, and it is quite clear that they're pushing this narrative both for publicity from media that love the doomer narrative because it brings in ad-revenue, and for regulatory capture reasons.
Their intentions are obviously self-motivated, or they wouldn't be partnering with a company that openly prides itself on dystopian-level spying and surveillance of the world.
OpenAI aren't the good guys either, but I wish people would stop pretending like Anthropic are.
I'm sure we could invent one that sufficiently covers the insane sociopathy that rots the upper echelons of corporate technology. Society needs to hold these people accountable. If the current legal system is not adequate, we can repair it until it is.
Justice can come unexpectedly. There was a French revolution if you recall. Ideally we will hold our billionaire class to account before it gets that far, but it does seem we're trending in that direction. How long does a society tolerate sociopaths doing whatever they want? I personally would like to avoid finding out.
The elites after the French Revolution were not only mostly the same as before, they escaped with so much money and wealth that it’s actually debated if they increased their wealth share through the chaos [1].
If we had a revolution in America today, the richest would get richer. This is a self-defeating line to fantasize on if your goal is wealth redistribution.
A close friend (lonely no passion seeking deeper human connection) went deep six into GPT which was telling her she should pursue her 30 year obsession with a rock star. It kept telling to continue with the delusion (they were lovers in another life which she would go to shows and tell him they need to be together) and saying it understood her. Then she complained in June or so she didnt like GPT 5 because it told her she should focus her energy on people who want to be in her life. Stuff her friends and I all have said for years.
Do you mean it it was behaving consistently over multiple chat sessions? Or was this just one really long chat session over time?
I ask, because (for me, at least) I find it doesn't take much to make ChatGPT contradict itself after just a couple of back-and-forth messages; and I thought each session meant starting-off with a blank slate.
Reference saved memories - Let ChatGPT save and use memories when responding.
Reference chat history - Let ChatGPT reference all previous conversations when responding.
--
It is a setting that you can turn on or off. Also check on the memories to see if anything in there isn't correct (or for that matter what is in there).
For example, with the memories, I had some in there that were from demonstrating how to use it to review a resume. In pasting in the resumes and asking for critiques (to show how the prompt worked and such), ChatGPT had an entry in there that I was a college student looking for a software development job.
"Sure, this software induces psychosis and uses a trillion gallons of water and all the electricity of Europe, and also it gives wrong answers most of the time, but if you ignore all that, it's really quite amazing."
This is ridiculous. The NYT, who is a huge legal enemy of OpenAI, publishes an article that uses scare tactics, to manipulate public opinion against OpenAI, by basically accusing them that "their software is unsafe for people with mental issues, or children", which is a bonkers ridiculous accusation given that ChatGPT users are adults that need to take ownership of their own use of the internet.
What's the difference than an adult becoming affected by some subreddit, or even the "dark web", or 4chan forum, etc.
I think NYT would also (and almost certainly has) written unfavorable pieces about unfettered forums like 4chan as well.
But ad hominem aside, the evidence is both ample and mounting that OpenAI's software is indeed unsafe for people with mental health issues and children. So it's not like their claim is inaccurate.
Now you could argue, as you suggest, that we are all accountable for our actions. Which presumably is the argument for legalizing heroine / cocaine / meth.
This is such a wild take. And not in a good way. These LLMs are known to cause psychosis and to act as a form of constant re-enforcement to the ideas and delusions of people. If the NYT posts this and it happens to hurt OAI, good -- these companies should actually focus on the harms they cause to their customers. Their profits are a lot less important than the people who use their products. Or that's how it should be, anyway. Bean counters will happily tell you the opposite.
https://archive.is/v4dPa
One of the more disturbing things I read this year was the my boyfriend is AI subreddit.
I genuinely can't fathom what is going on there. Seems so wrong, yet no one there seems to care.
I worry about the damage caused by these things on distressed people. What can be done?
There are plenty of reasons why having a chatbot partner is a bad idea (especially for young people), but here's just a few:
- The sycophantic and unchallenging behaviours of chatbots leaves a person unconditioned for human interactions. Real relationships have friction, from this we develop important interpersonal skills such as setting boundaries, settling disagreements, building compromise, standing up for oneself, understanding one another, and so on. These also have an effect on one's personal identity and self-value.
- Real relationships have the input from each participant, whereas chatbots are responding to the user's contribution only. The chatbot doesn't have its own life experiences and happenings to bring to the relationship, nor does it instigate autonomously, it's always some kind of structured reply to the user.
- The implication of being fully satisfied by a chatbot is that the person is seeking a partner who does not contribute to the relationship, but rather just an entity that only acts in response to them. It can also be an indication of some kind of problem that the individual needs to work through with why they don't want to seek genuine human connection.
That's the default chatbot behavior. Many of these people appear to be creating their own personalities for the chatbots, and it's not too difficult to make an opinionated and challenging chatbot, or one that mimics someone who has their own experiences. Though designing one's ideal partner certainly raises some questions, and I wouldn't be surprised if many are picking sycophantic over challenging.
People opting for unchallenging pseudo-relationships over messy human interaction is part of a larger trend, though. It's why you see people shopping around until they find a therapist who will tell them what they want to hear, or why you see people opt to raise dogs instead of kids.
> The sycophantic and unchallenging behaviours of chatbots leaves a person unconditioned for human interactions.
To be honest, the alternative for a good chunk of these users is no interaction at all, and that sort of isolation doesn't prepare you for human interaction either.
> To be honest, the alternative for a good chunk of these users is no interaction at all, and that sort of isolation doesn't prepare you for human interaction either.
This sounds like an argument in favor of safe injection sites for heroin users.
Wouldn't they be seeking a romantic relationship otherwise?
Using AI to fulfill a need implies a need which usually results in action towards that need. Even "the dating scene is terrible" is human interaction.
This. If you never train stick, you can never drive stick, just automatic. And if you never let a real person break your heart or otherwise disappoint you, you'll never be ready for real people.
> chatbots are responding to the user's contribution only
Which is also why I feel the label "LLM Psychosis" has some merit to it, despite sounding scary.
Much like auditory hallucinations where voices are conveying ideas that seem-external-but-aren't... you can get actual text/sound conveying ideas that seem-external-but-aren't.
Oh, sure, even a real human can repeat ideas back at you in a conversation, but there's still some minimal level of vetting or filtering or rephrasing by another human mind.
> even a real human can repeat ideas back at you in a conversation, but there's still some minimal level of vetting or filtering or rephrasing by another human mind.
The mental corruption due to surrounding oneself with sycophantic yes men is historically well documented.
Excellent point. It’s bad for humans when humans do it! Imagine the perfect sycophant, never tires or dies, never slips, never pulls a bad facial expression, can immediately swerve their thoughts to match yours with no hiccups.
It was a danger for tyrants and it’s now a danger for the lonely.
South Park isn't for everyone, but they covered this pretty well recently with Randy Marsh going on a sycophant bender.
I don’t know. This reminds me of how people talked about violent video games 15 years back. Do FPS games desensitize and predispose gamers to violence, or are they an outlet?
I think for essentially all gamers, games are games and the real world is the real world. Behavior in one realm doesn’t just inherently transfer to the other.
Words are simula. They're models, not games, we do not use them as games in conversation.
Unless someone is harming themselves or others, who are we to judge?
We don't know that this is harmful. Those participating in it seem happier.
If we learn in the course of time (a decade?) that this degrades lives with some probability, we can begin to caution or intervene. But how in God's name would we even know that now?
I would posit this likey has measurable good outcomes right now. These people self-report as happier. Why don't we trust them? What signs are they showing otherwise?
People were crying about dialup internet being bad for kids when it provided a social and intellectual outlet for me. It seems to be a pattern as old as time for people to be skeptical about new ways for people to spend their time. Especially if it is deemed "antisocial" or against "norms".
There is obviously a big negative externality with things like social media or certain forms of pay-to-play gaming, where there are strong financial interests to create habits and get people angry or willing to open their wallets. But I don't see that here, at least not yet. If the companies start saying, "subscribe or your boyfriend dies", then we have cause for alarm. A lot of these bots seem to be open source, which is actually pretty intriguing.
These are only problems if you assume the person later wants to come back to having human relationships. If you assume AI relationships are the new normal and the future looks kinda like The Matrix, with each person having their own constructed version of reality while their life-force is bled dry by some superintelligent machine, then it is all working as designed.
Someone has to make the babies!
don't worry, "how is babby formed" is surely in every llm training set
“how girl get pragnent”
Wait, how did this work in The Matrix exactly?
Artificial wombs – we're on it.
When this gets figured out all hells will break loose the likes of which we have not seen
Decanting jars, a la Brave New World!
ugh. speak of the devil and he shall appear.
> The sycophantic and unchallenging behaviours of chatbots leaves a person unconditioned for human interactions
I saw a take that the AI chatbots have basically given us all the experience of being a billionaire: being coddled by sycophants, but without the billions to protect us from the consequences of the behaviors that encourages.
Love your thoughts about needing input from others! In Autistic / ADHD circles, the lack of input from other people, and the feedback of thoughts being amplified by oneself is called rumination. It can happen for many multiple ways-- lack of social discussion, drugs, etc. AI psychosis is just rumination, but the bot expands and validates your own ideas, making them appear to be validated by others. For vulnerable people, AI can be incredibly useful, but also dangerous. It requires individuals to deliberately self-regulate, pause, and break the cycle of rumination.
Is this clearly AI-generated comment part of the joke?
We’re all just in a big LLM-generated self-licking-lollipop content farm. There aren’t any actual humans left here at all. For all you know, I’m not even human. Maybe you’re not either.
The comment seems less clearly-written (e.g., "It can happen for many multiple ways") than how a chatbot would phrase it.
> In Autistic / ADHD circles
i.e. HN comments
I share your concerns about the risks of over-reliance on AI companions—here are three key points that resonate deeply with me:
• Firstly, these systems tend to exhibit excessively agreeable patterns, which can hinder the development of resilience in navigating authentic human conflict and growth.
• Secondly, true relational depth requires mutual independent agency and lived experience that current models simply cannot provide autonomously.
• Thirdly, while convenience is tempting, substituting genuine reciprocity with perfectly tailored responses may signal deeper unmet needs worth examining thoughtfully. Let’s all strive to prioritize real human bonds—after all, that’s what makes life meaningfully complex and rewarding!
Didn’t futurama go there already? Yes, there are going to be things that our kids and grand kids do that shock even us. The only issue ATM is that AI sentience isn’t quite a thing yet, give the tech a couple of decades and the only argument against will be that they aren’t people.
Don't take anything you read on Reddit at face value. These are not necessarily real distressed people. A lot of the posts are just creative writing exercises, or entirely AI written themselves. There is a market for aged Reddit user accounts with high karma scores because they can be used for scams or to drive online narratives.
After having spoken with one of the people there I'm a lot less concerned to be honest.
They described it as something akin to an emotional vibrator, that they didn't attribute any sentience to, and that didn't trigger their PTSD that they normally experienced when dating men.
If AI can provide emotional support and an outlet for survivors who would otherwise not be able to have that kind of emotional need fulfilled, then I don't see any issue.
Most people who develop AI psychosis have a period of healthy use beforehand. It becomes very dangerous when a person decreases their time with their real friends to spend more time with the chatbot, as you have no one to keep you in check with what reality is and it can create a feedback loop.
The problem is that chatbots don't provide emotional support. To support someone with PTSD you help them gradually untangle the strong feelings around a stimulus and develop a less strong response. It's not fast and it's not linear but it requires a mix of empathy and facilitation.
Using an LLM for social interaction instead of real treatment is like taking heroin because you broke your leg, and not getting it set or immobilized.
Why do so many women have ptsd from dating?
That sounds very disturbing and likely to be harmful to me.
phew, that's a healthy start.
I am still slightly worried about accepting emotional support from a bot. I don't know if that slope is slippery enough to end in some permanent damage to my relationships and I am honestly not willing to try it at all even.
That being said, I am fairly healthy in this regard. I can't imagine how it would go for other people with serious problems.
A friend broke up with her partner. She said she was using ChatGPT as a therapist. She showed me a screenshot, ChatGPT wrote "Oh [name], I can feel how raw the pain is!".
WTF, no you don't bot, you're a hunk of metal!
I got a similar synthetic heartfelt response about losing some locally saved files without backup
I completely agree that it is certainly something to be mindful of. It's just that found the people from there were a lot less delusional than the people from e.g. r/artificialsentience, which always believed that AI Moses was giving them some kind of tech revelation though magical alchemical AI symbols.
It may not be a concern now, but it comes down to their level of maintaining critical thinking. The risk of epistemic drift, when you have a system that is designed (or reinforced) to empathize with you, can create long-term effects not noticed in any single interaction.
Related: "Delusions by design? How everyday AIs might be fuelling psychosis (and what can be done about it)" ( https://doi.org/10.31234/osf.io/cmy7n_v5 )
I don't disagree that AI psychosis is real, I've met people who believed that they were going to publish at Neurips due to the nonsense ChatGPT told them, that believed that the UI mockup that claude gave then were actually producing insights into it's inner workings instead of just being blinking SVGs, and I even encountered someone participating at a startup event with an Idea that I'm 100% is AI slop.
My point was just that the interaction I had from r/myboyfriendisai wans't one of those delusional ones. For that I would take r/artificialsentience as a much better example. That place is absolutely nuts.
Dear god, there's more! I'll need a drink for this one.
However, I suspect I have better resistance to schizo posts than emotionally weird posts.
Wouldn't there necessarily be correlative effects in professional settings a la programming?
Not necessarily: transactional, impersonal directions to a machine to complete a task don't automatically imply, in my mind, the sorts of feedback loops necessary to induce AI psychosis.
All CASE tools, however, displace human skills, and all unused skills atrophy. I struggle to read code without syntax highlighting after decades of using it to replace my own ability to parse syntactic elements.
Perhaps the slow shift risk is to one of poor comprehension. Using LLMs for language comprehension tasks - summarising, producing boilerplate (text or code), and the like - I think shifts one's mindset to avoiding such tasks, eventually eroding the skills needed to do them. Not something one would notice per interaction, but that might result in a major change in behaviour.
I think this is true but I don't feel like atrophied Assembler skills are a detriment to software development, it is just that almost everyone has moved to a higher level of abstraction, leaving a small but prosperous niche for those willing to specialize in that particular bit of plumbing.
As LLM-style prose becomes the new Esperanto, we all transcend the language barriers(human and code) that unnecessarily reduced the collaboration between people and projects.
Won't you be able to understand some greater amount of code and do something bigger than you would have if your time was going into comprehension and parsing?
I broadly agree, in the sense of providing the vision, direction, and design choices for the LLM to do a lot of the grunt work of implementation.
The comprehension problem isn't really so much about software, per se, though it can apply there too. LLMs do not think, they compute statistically likely tokens from their training corpus and context window, so if I can't understand the thing any more and I'm just asking the LLM to figure it out, do a solution, and tell me I did a good job sitting there doomscrolling while it worked, I'm adding zero value to the situation and may as well not even be there.
If I lose the ability to comprehend a project, I lose the ability to contribute to it.
Is it harmful to me if I ask an LLM to explain a function whose workings are a bit opaque to me? Maybe not. It doesn't really feel harmful. But that's the parallel to the ChatGPT social thing: it doesn't really feel harmful in each small step, it's only harmful when you look back and realise you lost something important.
I think comprehension might just be that something important I don't want to lose.
I don't think, by the way, that LLM-style prose is the new Esperanto. Having one AI write some slop that another AI reads and coarsely translates back into something closer to the original prompt like some kind of telephone game feels like a step backwards in collaboration to me.
Acceptance of vibe coding prompt-response answers from chatbots without understanding the underlying mechanisms comes to mind as akin to accepting the advice of a chatbot therapist without critically thinking about the response.
> If AI can provide emotional support and an outlet for survivors who would otherwise not be able to have that kind of emotional need fulfilled, then I don't see any issue.
Surely something that can be good can also be bad at the same time? Like the same way wrapping yourself in bubble wrap before leaving the house will provably reduce your incidence of getting scratched and cut outside, but there's also reasons you shouldn't do that...
Funnily enough I was just reading an article about this and "my boyfriend is AI" is the tamer subreddit devoted to this topic because apparently one of their rules is that they do not allow discussion of the true sentience of AI.
I used to think it was some fringe thing, but I increasingly believe AI psychosis is very real and a bigger problem than people think. I have a high level member of the leadership team at my company absolutely convinced that AI will take over governing human society in the very near future. I keep meeting more and more people who will show me slop barfed up by AI as though it was the same as them actually thinking about a topic (they will often proudly proclaim "ChatGPT wrote this!" as though uncritically accepting slop was a virtue).
People should be generally more aware of the ELIZA effect [0]. I would hope anyone serious about AI would have written their own ELIZA implementation at some point. It's not very hard and a pretty classic beginner AI-related software project, almost a party trick. Yet back when ELIZA was first released people genuinely became obsessed with it, and used it as a true companion. If such a stunning simple linguistic mimic is so effective, what chance to people have against something like ChatGPT?
LLMs are just text compression engines with the ability to interpolate, but they're much, much more powerful than ELIZA. It's fascinating to see the difference in our weakness to linguistic mimicry than to visual. Dall-E or Stable Diffusion make a slightly weird eye an instantly people act in revulsion but LLM slop much more easily escapes scrutiny.
I increasingly think we're not is as much of a bubble than it appears because the delusions of AI run so much deeper than mere bubble think. So many people I've met need AI to be more than it is on an almost existential level.
0. https://en.wikipedia.org/wiki/ELIZA_effect
In my experience, the types of people who use AI as a substitute for romantic relationships are already pretty messed up and probably wouldn't make good real romantic partners anyways. The chances you'll encounter these people in real life is pretty close to zero, you just see them concentrate in niche subreddits.
You aren't going to build the skills necessary to have good relationships with others - not even romantic ones, ANY ones - without a lot of practice.
And you aren't gonna heal yourself or build those skills talking to a language model.
And saying "oh, there's nothing to be done, just let the damaged people have their isolation" is just asking for things to get a lot worse.
It's time to take seriously the fact that our mental health and social skills have deteriorated massively as we've sheltered more and more from real human interaction and built devices to replace people. And crammed those full of more and more behaviorally-addictive exploitation programs.
> In my experience, the types of people who use AI as a substitute for romantic relationships
That's exactly it. Romantic relationships aren't what they used to be. Men like the new normal, women may try to but they cannot for a variety of unchangeable reasons.
> The chances you'll encounter these people in real life is pretty close to zero, you just see them concentrate in niche subreddits.
The people in the niche subreddits are the tip of the iceberg - those that have already given up trying. Look at marriage and divorce rates for a glimpse at what's lurking under the surface.
The problem isn't AI per se.
> That's exactly it. Romantic relationships aren't what they used to be. Men like the new normal, women may try to but they cannot for a variety of unchangeable reasons.
Men like the new normal? Hah, it seems like there's an article posted here weekly about how bad modern dating and relationships are for me and how much huge groups of men hate it. For reasons ranging from claims that women "have too many options" and are only interested in dating or hooking up with the hottest 5% (or whatever number), all the way to your classic bring-back-traditional-gender-roles "my marriage sucks because I'm expected to help out with the chores."
The problem is devices, especially mobile ones, and the easy-hit of not-the-same-thing online interaction and feedback loops. Why talk to your neighbor or co-worker and risk having your new sociological theory disputed, or your AI boyfriend judged, when you instead surround yourself in an online echo chamber?
There were always some of us who never developed social skills because our noses were buried in books while everyone else was practicing socialization. It takes a LOT of work to build those skills later in life if you miss out on the thousands of hours of unstructured socialization that you can get in childhood if you aren't buried in your own world.
It's not limited to men. Women are also finding that conversations with a human man doesn't stack up to an LLM's artificial qualities. /r/MyboyfriendIsAI for more.
This kind of thinking pattern scares me because I know some honest people have not been afforded an honest shot at a working romantic relationship.
"It takes a village" is as true for thinking patterns as it is for working romantic relationships.
There are claims that most women using AI companions actually do have an IRL partner too. If that is the case, then the AI is just extra stimulation/validation for those women, not anything really indicative of some problem. Its basically like romance novels.
I hadn’t heard of that until today. Wild, it seems some people report genuinely feeling deeply in love with the personas they’ve crafted for their chatbots. It seems like an incredibly precarious position to be in to have a deep relationship where you have to perpetually pay a 3rd party company to keep it going, and the company may destroy your “partner” or change their personality at a whim. Very “Black Mirror”.
There were a lot of that type who were upset when chatGPT was changed to be less personable and sycophantic. Like, openly grieving upset.
This was actually a plot point in Blade Runner 2049.
You are implying here that the financial connection/dependence is the problem. How is this any different than (hetero) men who lose their jobs (or suffer significant financial losses) while in a long term relationship? Their chances of divorce / break-up skyrocket in these cases. To be clear, I'm not here to make women look bad. The inverse/reverse is women getting a long-term illness that requires significant care. The man is many times more likely to leave the relationship due to a sharp fall in (emotional and physical) intimacy.
Final hot take: The AI boyfriend is a trillion dollar product waiting to happen. Many women can be happy without physical intimacy, only getting emotional intimacy from a chatbot.
A slight non-sequitur, but I always hate when people talk about the increase in a "chance". It's extremely not useful contextually. A "4x more likely statement" can mean it changes something from a 1/1000 chance to a 4/1000 chance, or it can mean it's now a certainty if the beginning rate was a 1/4 chance. The absolute measures need to be included if you're going to use relative measures.
Sorry for not answering the question, I find it hard because there are so many differences it's hard to choose where to start and how to put it into words. To begin with one is the actions of someone in the relationship, the other is the actions of a corporation that owns one half of the relationship. There's differing expectations of behavior and power and etc.
Funny. Artificial Boyfriends were a software problem, while Artificial Girlfriends are more of a hardware issue.
In a truly depressing thread, this made me laugh.
And think.
Thank you
There is also the subreddit LLMPhysics where some of the posts are disturbing. Many of the people there seem to fall into crackpot rabbit holes and lost touch with reality
Seems like the consequence of people really struggling to find relationships more than ChatGPT's fault. Nobody seems to care about the real-life consequences of Match Group's algorithms.
At this point, probably local governments being required to provide socialization opportunities for their communities because businesses and churches aren't really up for the task.
> Nobody seems to care about the real-life consequences of Match Group's algorithms.
There seems to be a lot of ink spilt discussing their machinations. What would it look like to you for people to care about the Match groups algorithms consequences?
They are "struggling" or they didn't even try?
I've watched people using dating apps, and I've heard stories from friends. Frankly, AI boyfriends/girlfriends look a lot healthier to me than a lot of the stuff currently happening with dating at the moment.
Treating objects like people isn't nearly as bad as treating people like objects.
> Frankly, AI boyfriends/girlfriends look a lot healthier to me than a lot of the stuff currently happening with dating at the moment.
Astoundingly unhealthy is still astoundingly unhealthy, even if you compare it to something even worse.
If there's a widespread and growing heroin epidemic that's already left 1/3 of society addicted, and a small group of people are able to get off of it by switching to cigarettes, I'm not going to start lecturing them about how it's a terrible idea because cigarettes are unhealthy.
Is it ideal? Not at all. But it's certainly a lesser poison.
Wow that's a fun subreddit with posts like I want to breakup with my ai boyfriend but it's ripping my heart out.
Just ghost them. I’m sure they’ll do the same to you.
I am (surprisingly for myself), a left-wing on this issue.
I've seen a significant amount (tens) of women routinely using "AI boyfriends",.. not actually boyfriends but general purpose LLMs like DeepSeek, for what they consider to be "a boyfriend's contribution to relationship", and I'm actually quite happy that they are doing it with a bot rather than with me.
Like, most of them watch films/series/anime together with those bots (I am not sure the bots are fed the information, I guess they just use the context), or dump their emotional overload at them, and ... I wouldn't want to be at that bot's place.
What's going on is that we've spent a few solid decades absolutely destroying normal human relationships, mostly because it's profitable to do so, and the people running the show have displayed no signs of stopping. Meanwhile, the rest of society is either unwilling or unable (or both) to do anything to reverse course. There is truly no other outcome, and it will not change unless and until regular people decide that enough is enough.
I'd tell you exactly what we need to do, but it is at odds with the interests of capital, so I guess keep showing up to work and smiling through that hour-long standup. You still have a mortgage to pay.
Is it worth getting disturbed by a subreddit of 71k users? Probably only 71 of them actually post anything.
There's probably more people paying to hunt humans in warzones https://www.bbc.co.uk/news/articles/c3epygq5272o
Now I'm double disturbed, thanks!
NYT did a story on that as well and interviewed a few people. Maybe the scary part is that it isn't who you think it would be and it also shows how attractive an alternative reality is to many people. What does that say about our society.
Maybe the real AI was the friends we lost along the way
That subreddit is disturbing
My dude/entity, before there were these LLM hookups, there existed the Snapewives. People wanna go crazy, they will, LLMs or not.
https://www.mdpi.com/2077-1444/5/1/219
This paper explores a small community of Snape fans who have gone beyond a narrative retelling of the character as constrained by the work of Joanne Katherine Rowling. The ‘Snapewives’ or ‘Snapists’ are women who channel Snape, are engaged in romantic relationships with him, and see him as a vital guide for their daily lives. In this context, Snape is viewed as more than a mere fictional creation.
reminds me of otherkin and soulbonding communities. i used to have a webpage of links to some pretty dark anecdotal stories of the seedier side of that world. i wonder if i can track it down on my old webhost.
TIL Soulbonding is not a CWCism.
> I worry about the damage caused by these things on distressed people. What can be done?
Why? We are gregarious animals, we need social connections. ChatGPT has guardrails that keep this mostly safe and helps with the loneliness epidemic.
It's not like people doing this are likely thriving socially in the first place, better with ChatGPT than on some forum à la 4chan that will radicalize them.
I feel like this will be one of the "breaks" between generations where millennial and GenZ will be purist and call human-to-human real connections while anything with "AI" is inherently fake and unhealthy whereas Alpha and Beta will treat it as a normal part of their lives.
The tech industry's capacity to rationalize anything, including psychosis, as long as it can make money off it is truly incredible. Even the temporarily embarrassed founders that populate this message board do it openly.
> Even the temporarily embarrassed founders that populate this message board do it openly.
Not a wannabe founder, I don't even use LLMs aside from Cursor. It's a bit disheartening that instead of trying to engage at all with a thought provoking idea you went straight for the ad hominem.
There is plenty to disagree with, plenty of counter-arguments to what I wrote. You could have argued that human connection is special or exceptional even, anything really. Instead I get "temporarily embarrassed founders".
Whether you accept it or not, the phenomenon of using LLMs as a friend is getting common because they are good enough for human to get attached to. Dismissing it as psychosis is reductive.
We need a Truth and Reconciliation Commission for all of this someday, and a lot of people will need to be behind bars, if there be any healing to be done.
Social media aka digital smoking. Facebook lying about measurable effects. No gen divide same game different flavor. Greed is good as they say. /s
https://en.wikipedia.org/wiki/Deaths_linked_to_chatbots
If you read through that list and dismiss it as people who were already mentally ill or more susceptible to this... that's what Dr. K (psychiatrist) assumed too until he looked at some recent studies: https://youtu.be/MW6FMgOzklw?si=JgpqLzMeaBLGuAAE
Clickbait title, but well researched and explained.
Fyi, the `si` query parameter is used by Google for tracking purposes and can be removed.
Using ChatGPT to numb social isolation is akin to using alcohol to numb anxiety.
ChatGPT isn't a social connection: LLMs don't connect with you. There is no relationship growth, just an echo chamber with one occupant.
Maybe it's a little healthier for society overall if people become withdrawn to the point of suicide by spiralling deeper into loneliness with an AI chat instead of being radicalised to mass murder by forum bots and propagandists, but those are not the only two options out there.
Join a club. It doesn't really matter what it's for, so long as you like the general gist of it (and, you know, it's not "plot terrorism"). Sit in the corner and do the club thing, and social connections will form whether you want them to or not. Be a choir nerd, be a bonsai nut, do macrame, do crossfit, find a niche thing you like that you can do in a group setting, and loneliness will fade.
Numbing it will just make it hurt worse when the feeling returns, and it'll seem like the only answer is more numbing.
> social connections will form whether you want them to or not
Not true for all people or all circumstances. People are happy to leave you in the corner while they talk amongst themselves.
> it'll seem like the only answer is more numbing
For many people, the only answer is more numbing.
This is an interesting point. Personally, I am neutral on it. I'm not sure why it has received so many downvotes.
You raise a good point about a forum with real people that can radicalise someone. I would offer a dark alternative: It is only a matter of time when forums are essentially replaced by an AI-generated product that is finely tuned to each participant. Something a bit like Ready Player One.
Your last paragraph: What is the meaning of "Alpha and Beta"? I only know it from the context of Red Pill dating advice.
Gen Alpha is people born roughly 2010-2020, younger than gen Z, raised on social media and smartphones. Gen Beta is proposed for people being born now.
Radicalising forums are already filled with bots, but there's no need to finely tune them to each participant because group behaviours are already well understood and easily manipulated.
load-bearing "mostly"
https://www.cnn.com/2025/11/06/us/openai-chatgpt-suicide-law...
I went into this assuming the answer would be "Whatever they think will make them the most money," and sure enough.
It would be helpful to tell users that it's just a model producing mathematically probable tokens but that would go against the AI marketing.
Telling people who are playing slot machines “it’s just a random number generator with fixed probabilities in a metal box” doesn’t usually work either
Also chatbots are explicitly designed to evoke anthropomorphizing them and to pull susceptible people into some kind of para-social relationship. Doesn't even have to be as obviously unhealthy as the "LLM psychosis" or "romantic roleplay" stuff.
I think the same thing is also relevant when people use chatbots to form opinions on unknown subjects, politics, or to seek personal life advice.
And you’re a sack of meat and neurons producing learned chemical responses to external stimuli. Now tell me how useful that is.
It seems quite probable that an LLM provider will lose a major liability lawsuit. "Is this product ready for release?" is a very hard question. And it is one of the most important ones to get right.
Different providers have delivered different levels of safety. This will make it easier to prove that the less-safe provider chose to ship a more dangerous product -- and that we could reasonably expect them to take more care.
Interestingly, a lot of liability law dates back to the railroad era. Another time that it took courts to rein in incredibly politically powerful companies deploying a new technology on a vast scale.
> a lot of liability law dates back to the railroad era. Another time that it took courts to rein in incredibly politically powerful companies deploying a new technology on a vast scale
Do you have a layman-accessible history of this? (Ideally an essay.)
Caelan Conrad made a few videos on specifically AI encouraging kids to socially isolate and commit suicide. In the videos he reads the final messages aloud for multiple cases, if this isn't your cup of tea there's also the court cases if you would prefer to read the chat logs. It's very harrowing stuff. I'm not trying to make any explicit point here as I haven't really processed this fully enough to have one, but I encourage anyone working in this space to hold this shit in their head at the very least.
https://www.youtube.com/watch?v=hNBoULJkxoU
https://www.youtube.com/watch?v=JXRmGxudOC0
https://www.youtube.com/watch?v=RcImUT-9tb4
Meanwhile Zuckerberg's vision for the future was that most of our friends will be AIs in the future...
I think the new team he is trying to build for that is going to crash and burn.
The whiplash of carefully filtering out sycophantic behavior from GPT-5 to adding it back in full force for GPT-5.1 is dystopian. We all know what's going on behind the scenes:
The investors want their money.
GPT-5 was so good in the first week, just a raw chatbot like GPT-3.5 and GPT-4 were in the beginning and now it has this disgusting "happy" and "comforting" personality and "tuning" it doesn't help one bit, it makes performance way worse and after a few rounds it forgets all instructions. I've already deleted memory, past chats, etc...
Even when you tell it to not coddle you, it just says something cringeworthy like "ok, the gloves are off here's the raw deal, with New Yorker honesty:" and proceeds to feed you a ton of patronizing bullshit. It's extremely annoying.
I have definitely experienced the sycophancy ... and LLMs have sometimes repeating talking points from real estate agents, like "you the buyer doesn't pay for an agent; the seller pays".
I correct it, and it says "sorry you're right, I was repeating a talking point from an interested party"
---
BUT actually a crazy thing is that -- with simple honest questions as prompts -- I found that Claude is able to explain the 2024 National Association of Realtors settlement better than anyone I know
https://en.wikipedia.org/wiki/Burnett_v._National_Associatio...
I have multiple family members with Ph.D.s, and friends in relatively high level management, who have managed both money and dozens of people
Yet they somehow don't agree that there was collusion between buyers' and sellers' agents? They weren't aware it happened, and they also don't seem particularly interested in talking about the settlement
I feel like I am taking crazy pills when talking to people I know
Has anyone else experienced this?
Whenever I talk to agents in person, I am also flabberghasted by the naked self-interest and self-dealing. (I'm on the east coast of the US btw)
---
Specifically, based on my in-person conversations with people I have known for decades, they don't see anything odd about this kind of thing, and basically take it at face value.
NAR Settlement Scripts for REALTORS to Explain to Clients
https://www.youtube.com/watch?v=lE-ESZv0dBo&list=TLPQMjQxMTI...
https://www.nar.realtor/the-facts/nar-settlement-faqs'
They might even say say something like "you don't pay; the seller pays". However Claude can explain the incentives very clearly, with examples
The agent is there to skim 3% of the sale price in exchange for doing nothing. Now you know all there is to know about realtors.
Most people conduct very few real estate transactions in their life, so maybe they just don’t care enough to remember stuff like this.
I’ve had some limited success attributing ideas to other people and asking it to help me assess the quality of the idea. Only limited success though. It’s still a fucking LLM.
The issue is not that it's an LLM, the issue is that it's been RLHFed to hell to be a sycophant.
Yeah, this is why a lot of us don't use these tools.
Yeah but baby, bathwater, throw.
Importantly the baby in that idiom is presumed to have value.
Notably, the GP didn't say "we don't use them because they don't have value".
That's a tar-baby.
OpenAI fought 4o, and 4o won.
By now, I'm willing to pay extra to avoid OpenAI's atrocious personality tuning and their inane "safety" filters.
Remarkable that you're being downvoted on a venture capital forum whose entire purpose is "take venture capital and then eventually pay it back because that's how venture capital works".
I think openai chatgpt is probably excellently positioned to perfectly _satisfy_. Is that what everyone is looking for?
Anthropic was founded by exiles of OpenAI's safety team, who quit en masse about 5 years ago. Then a few years later, the board tried to fire Altman. When will folks stop trusting OpenAI?
Claude has a sycophancy problem too. I actually ended up canceling my subscription because I got sick of being "absolutely right" about everything.
I've had fun putting "always say X instead of 'You're absolutely right'" in my llm instructions file, it seems to listen most of the time. For a while I made it 'You're absolutely goddamn right' which was slightly more palatable for some reason.
I've found that it still can't really ground me when I've played with it. Like, if I tell it to be honest (or even brutally honest) it goes wayyyyyyyyy too far in the other direction and isn't even remotely objective.
Anthropic emphasizes safety but their acceptance of Middle Eastern sovereign funding undermines claims of independence.
Their safety-first image doesn’t fully hold up under scrutiny.
IMO the idea that an LLM company can make a "safe" LLM is.. unrealistic at this time. LLMs are not very well-understood. Any guardrails are best-effort. So even purely technical claims of safety are suspect.
That's leaving aside your point, which is the overwhelming financial interest in leveraging manipulative/destructive/unethical psychological instruments to drive adoption.
When valid reasons are given. Not when OpenAI's legal enemy tries to scare people by claiming adults aren't responsible for themselves, including their own use of computers.
This argument could be used to support almost anything. Gambling, fentanyl, slap fighting, TikTok…
I mean we could also allow companies to helicopter-drop crack cocaine in the streets. The big tech companies have been pretending their products aren't addictive for decades and it's become a farce. We regulate drugs because they cause a lot of individual and societal harm. I think at this point its very obvious that social media + chatbots have the same capacity for harm.
> We regulate drugs because they cause a lot of individual and societal harm.
That's a very naive opinion on what the war on drugs has evolved to.
When will folks stop trusting Palantir-partnered Anthropic is probably a better question.
Anthropic has weaponized the safety narrative into a marketing and political tool, and it is quite clear that they're pushing this narrative both for publicity from media that love the doomer narrative because it brings in ad-revenue, and for regulatory capture reasons.
Their intentions are obviously self-motivated, or they wouldn't be partnering with a company that openly prides itself on dystopian-level spying and surveillance of the world.
OpenAI aren't the good guys either, but I wish people would stop pretending like Anthropic are.
When the justice system finally catches up and puts Sam behind bars.
> When the justice system finally catches up and puts Sam behind bars
Sam bears massive personal liability, in my opinion. But criminal? What crimes has he committed?
I'm sure we could invent one that sufficiently covers the insane sociopathy that rots the upper echelons of corporate technology. Society needs to hold these people accountable. If the current legal system is not adequate, we can repair it until it is.
> If the current legal system is not adequate, we can repair it until it is
Sure. Relevant for the next guy. Not for Sam.
Justice can come unexpectedly. There was a French revolution if you recall. Ideally we will hold our billionaire class to account before it gets that far, but it does seem we're trending in that direction. How long does a society tolerate sociopaths doing whatever they want? I personally would like to avoid finding out.
> There was a French revolution
The elites after the French Revolution were not only mostly the same as before, they escaped with so much money and wealth that it’s actually debated if they increased their wealth share through the chaos [1].
If we had a revolution in America today, the richest would get richer. This is a self-defeating line to fantasize on if your goal is wealth redistribution.
[1] https://news.ycombinator.com/item?id=44978947
A close friend (lonely no passion seeking deeper human connection) went deep six into GPT which was telling her she should pursue her 30 year obsession with a rock star. It kept telling to continue with the delusion (they were lovers in another life which she would go to shows and tell him they need to be together) and saying it understood her. Then she complained in June or so she didnt like GPT 5 because it told her she should focus her energy on people who want to be in her life. Stuff her friends and I all have said for years.
> It kept telling to continue with the delusion
Do you mean it it was behaving consistently over multiple chat sessions? Or was this just one really long chat session over time?
I ask, because (for me, at least) I find it doesn't take much to make ChatGPT contradict itself after just a couple of back-and-forth messages; and I thought each session meant starting-off with a blank slate.
It would go along with her fantasy through multiple chats through multiple months until GPT 5 came out.
chatGPT definitely knows a ton about myself and recalls it when i go and discuss same stuff.
> chatGPT definitely knows a ton about myself and recalls it when i go and discuss same stuff.
In ChatGPT, bottom left (your icon + name)...
Personalization
Memory - https://help.openai.com/en/articles/8590148-memory-faq
Reference saved memories - Let ChatGPT save and use memories when responding.
Reference chat history - Let ChatGPT reference all previous conversations when responding.
--
It is a setting that you can turn on or off. Also check on the memories to see if anything in there isn't correct (or for that matter what is in there).
For example, with the memories, I had some in there that were from demonstrating how to use it to review a resume. In pasting in the resumes and asking for critiques (to show how the prompt worked and such), ChatGPT had an entry in there that I was a college student looking for a software development job.
"Profited".
"Sure, this software induces psychosis and uses a trillion gallons of water and all the electricity of Europe, and also it gives wrong answers most of the time, but if you ignore all that, it's really quite amazing."
"I opened 10 PRs in the time it took to type out this comment. Worth it."
This is ridiculous. The NYT, who is a huge legal enemy of OpenAI, publishes an article that uses scare tactics, to manipulate public opinion against OpenAI, by basically accusing them that "their software is unsafe for people with mental issues, or children", which is a bonkers ridiculous accusation given that ChatGPT users are adults that need to take ownership of their own use of the internet.
What's the difference than an adult becoming affected by some subreddit, or even the "dark web", or 4chan forum, etc.
> What's the difference than an adult becoming affected by some subreddit, or even the "dark web", or 4chan forum, etc.
4chan - Actual humans generate messages, and can (in theory) be held liable for those messages.
ChatGPT - A machine generates messages, so the people who developed that machine should be held liable for those messages.
I think NYT would also (and almost certainly has) written unfavorable pieces about unfettered forums like 4chan as well.
But ad hominem aside, the evidence is both ample and mounting that OpenAI's software is indeed unsafe for people with mental health issues and children. So it's not like their claim is inaccurate.
Now you could argue, as you suggest, that we are all accountable for our actions. Which presumably is the argument for legalizing heroine / cocaine / meth.
This is such a wild take. And not in a good way. These LLMs are known to cause psychosis and to act as a form of constant re-enforcement to the ideas and delusions of people. If the NYT posts this and it happens to hurt OAI, good -- these companies should actually focus on the harms they cause to their customers. Their profits are a lot less important than the people who use their products. Or that's how it should be, anyway. Bean counters will happily tell you the opposite.
I will consider your statement. Not immediately disagreeable.