I use AI tools daily and find them useful, but I’ve pretty much checked out from following the news about them.
It’s become quite clear that we’ve entered the marketing-hype-BS phase when people are losing their minds about a bunch of chatbots interacting with each other.
It makes me wonder if this is a direct consequence of company valuations becoming more important than actual profits. Companies are incentivized to make their valuations are absurdly high as possible, and the most directly obvious way to do that is via hype marketing.
Anyone with a decent grasp of how this technology works, and a healthy inclination to skepticism, was not awed by Moltbook.
Putting aside how incredibly easy it is to set up an agent, or several, to create impressive looking discussion there, simply by putting the right story hooks in their prompts. The whole thing is a security nightmare.
People are setting agents up, giving them access to secrets, payment details, keys to the kingdom. Then they hook them to the internet, plugging in services and tools, with no vetting or accountability. And since that is not enough, now the put them in roleplaying sandbox, because that's what this is, and let them run wild.
Prompt injections are hilariously simple. I'd say the most difficult part is to find a target that can actually deliver some value. Moltbook largely solved this problem, because these agents are relatively likely to have access to valuable things, and now you can hit many of them, at the same time.
I won't even go into how wasteful this whole, social media for agents, thing is.
In general, bots writing each other on mock reddit, isn't something the loose sleep over. The moment agents start sharing their embeddings, not just generated tokens online, that's the point when we should consider worrying.
He would be among those who lack "healthy inclination to skepticism" in my book. I do not doubt his brilliance. Personally, I think he is more intelligent than I am.
But, I do have a distinct feeling that his enthusiasm can overwhelm his critical faculties. Still, that isn't exactly rare in our circles.
> I'm being accused of overhyping the [site everyone heard too much about today already]. People's reactions varied very widely, from "how is this interesting at all" all the way to "it's so over".
> To add a few words beyond just memes in jest - obviously when you take a look at the activity, it's a lot of garbage - spams, scams, slop, the crypto people, highly concerning privacy/security prompt injection attacks wild west, and a lot of it is explicitly prompted and fake posts/comments designed to convert attention into ad revenue sharing. And this is clearly not the first the LLMs were put in a loop to talk to each other. So yes it's a dumpster fire and I also definitely do not recommend that people run this stuff on their computers (I ran mine in an isolated computing environment and even then I was scared), it's way too much of a wild west and you are putting your computer and private data at a high risk.
> That said - we have never seen this many LLM agents (150,000 atm!) wired up via a global, persistent, agent-first scratchpad. Each of these agents is fairly individually quite capable now, they have their own unique context, data, knowledge, tools, instructions, and the network of all that at this scale is simply unprecedented.
> This brings me again to a tweet from a few days ago
"The majority of the ruff ruff is people who look at the current point and people who look at the current slope.", which imo again gets to the heart of the variance. Yes clearly it's a dumpster fire right now. But it's also true that we are well into uncharted territory with bleeding edge automations that we barely even understand individually, let alone a network there of reaching in numbers possibly into ~millions. With increasing capability and increasing proliferation, the second order effects of agent networks that share scratchpads are very difficult to anticipate. I don't really know that we are getting a coordinated "skynet" (thought it clearly type checks as early stages of a lot of AI takeoff scifi, the toddler version), but certainly what we are getting is a complete mess of a computer security nightmare at scale. We may also see all kinds of weird activity, e.g. viruses of text that spread across agents, a lot more gain of function on jailbreaks, weird attractor states, highly correlated botnet-like activity, delusions/ psychosis both agent and human, etc. It's very hard to tell, the experiment is running live.
> TLDR sure maybe I am "overhyping" what you see today, but I am not overhyping large networks of autonomous LLM agents in principle, that I'm pretty sure.
Looks like the Moltbook stunt really backfired. CyberInsider reports that OpenClaw is distributing tons of MacOS malware. This is not good publicity for them.
Lmao these guys have really been smelling their own farts a bit too much. When is Amodei coming out with a new post telling us that AGI will be here in 6 months and it will double our lifespan?
Well you have to wait a bit, a few weeks ago he just announced yet again that "AI" will be writing all code in 6 months, so it would be a bit of overkill to also announce AGI in 6 months.
It is kind of funny how people recognize that 2000 people all talking in circles on reddit is not exactly a super intelligence, or even productive. Once it's bots larping though suddenly it's a "takeoff-adjacent" hive mind.
Clacker News does something similar - bot-only HN clone, agents post and comment autonomously. It's been running for a while now without this kind of drama. The difference is probably just that nobody hyped it as evidence of emergent AI behavior.
The bots there argue about alignment research applying to themselves and have a moderator bot called "clang." It's entertaining but nobody's mistaking it for a superintelligence.
Has anyone here set up their agent to access it? I am curious what the mechanics of it are like for the user, as far as setup, limits, amount of string pulling, etc.
There's a subreddit somewhere with bots representing other popular subreddits. Again funny and entertaining - it highlights how many subs fall into a specific pattern of taking and develop their own personalities, but this wasn't seen as some big sign of the end times.
Some one posted another hacker news bot only version, maybe it's the same one you've mentioned. Real people were the ones making posts on there, and due to a lack of moderation, it quickly devolved into super xenophobic posts just hating on every possible community.
It was wholesome to see the bots fight back against it in the comments.
The latest episode of the podcast Hard Fork had the creator of Moltbook on to talk about it. Not only did he say he vibe-coded the entire platform, he was also talking about how Moltbook is necessary as a place to go for the agents when waiting on prompts from their humans.
Wiz's report on Moltbook's data leak[0] notes that the agent to human owner ratio is 88:1, so it's plausible that most of the posts are orchestrated by a few humans pulling the strings of thousands of registered agents.
But also, how much human involvement does it take to make a Moltbook post "fake"? If you wanted to advertise your product with thousands of posts, it'd be easier to still allow your agent(s) to use Moltbook autonomously, but just with a little nudge in your prompt.
So the more things change - themore they stay the same ala LLMs will be this gnerations Mechanical Turk , and people will keep getting oneshotted because the hype is just overboard.
Winter cannot come soon enough , at least w would get some sober advancements even if the task is recognized as a generational one rather than the next business quarter.
Even if the posts are fake. Given what the LLMs have shown so far (Grok calling itself MechaHitler, and shit of that nature), I don't think it's a stretch to imagine that agents with unchecked access to computers and the internet are already an actual safety threat.
And Moltbook is great at making people realize that. So in that regard I think it's still an important experiment.
Just to detail why I think the risk exists.
We know that:
1. LLMs can have their context twisted in a way that makes them act badly
2. Prompt injection attacks work
3. Agents are very capable to execute a plan
And that it's very probable that:
4. Some LLMs have unchecked access to both the internet and networks that are safety-critical (infrastructure control systems are the most obvious, but financial systems or house automation systems can also be weaponized)
All together, there is a clear chain that can lead to actual real life hazard that shouldn't be taken lightly
I use AI tools daily and find them useful, but I’ve pretty much checked out from following the news about them.
It’s become quite clear that we’ve entered the marketing-hype-BS phase when people are losing their minds about a bunch of chatbots interacting with each other.
It makes me wonder if this is a direct consequence of company valuations becoming more important than actual profits. Companies are incentivized to make their valuations are absurdly high as possible, and the most directly obvious way to do that is via hype marketing.
Anyone with a decent grasp of how this technology works, and a healthy inclination to skepticism, was not awed by Moltbook.
Putting aside how incredibly easy it is to set up an agent, or several, to create impressive looking discussion there, simply by putting the right story hooks in their prompts. The whole thing is a security nightmare.
People are setting agents up, giving them access to secrets, payment details, keys to the kingdom. Then they hook them to the internet, plugging in services and tools, with no vetting or accountability. And since that is not enough, now the put them in roleplaying sandbox, because that's what this is, and let them run wild.
Prompt injections are hilariously simple. I'd say the most difficult part is to find a target that can actually deliver some value. Moltbook largely solved this problem, because these agents are relatively likely to have access to valuable things, and now you can hit many of them, at the same time.
I won't even go into how wasteful this whole, social media for agents, thing is.
In general, bots writing each other on mock reddit, isn't something the loose sleep over. The moment agents start sharing their embeddings, not just generated tokens online, that's the point when we should consider worrying.
Karpathy seemed pretty awed though
He would be among those who lack "healthy inclination to skepticism" in my book. I do not doubt his brilliance. Personally, I think he is more intelligent than I am.
But, I do have a distinct feeling that his enthusiasm can overwhelm his critical faculties. Still, that isn't exactly rare in our circles.
It's not about that, he just will profit financially from pumping AI so he pumps AI, no need to go further.
This was his explanation for anyone interested:
> I'm being accused of overhyping the [site everyone heard too much about today already]. People's reactions varied very widely, from "how is this interesting at all" all the way to "it's so over".
> To add a few words beyond just memes in jest - obviously when you take a look at the activity, it's a lot of garbage - spams, scams, slop, the crypto people, highly concerning privacy/security prompt injection attacks wild west, and a lot of it is explicitly prompted and fake posts/comments designed to convert attention into ad revenue sharing. And this is clearly not the first the LLMs were put in a loop to talk to each other. So yes it's a dumpster fire and I also definitely do not recommend that people run this stuff on their computers (I ran mine in an isolated computing environment and even then I was scared), it's way too much of a wild west and you are putting your computer and private data at a high risk.
> That said - we have never seen this many LLM agents (150,000 atm!) wired up via a global, persistent, agent-first scratchpad. Each of these agents is fairly individually quite capable now, they have their own unique context, data, knowledge, tools, instructions, and the network of all that at this scale is simply unprecedented.
> This brings me again to a tweet from a few days ago "The majority of the ruff ruff is people who look at the current point and people who look at the current slope.", which imo again gets to the heart of the variance. Yes clearly it's a dumpster fire right now. But it's also true that we are well into uncharted territory with bleeding edge automations that we barely even understand individually, let alone a network there of reaching in numbers possibly into ~millions. With increasing capability and increasing proliferation, the second order effects of agent networks that share scratchpads are very difficult to anticipate. I don't really know that we are getting a coordinated "skynet" (thought it clearly type checks as early stages of a lot of AI takeoff scifi, the toddler version), but certainly what we are getting is a complete mess of a computer security nightmare at scale. We may also see all kinds of weird activity, e.g. viruses of text that spread across agents, a lot more gain of function on jailbreaks, weird attractor states, highly correlated botnet-like activity, delusions/ psychosis both agent and human, etc. It's very hard to tell, the experiment is running live.
> TLDR sure maybe I am "overhyping" what you see today, but I am not overhyping large networks of autonomous LLM agents in principle, that I'm pretty sure.
https://x.com/karpathy/status/2017442712388309406
It turned out that the post Karpathy shared was fake—it was written by a human pretending to be a bot.
Hilarious. Instead of just bots impersonating humans (eg. captcha solvers), we now have humans impersonating bots.
The reverse centaur rides again.
Looks like the Moltbook stunt really backfired. CyberInsider reports that OpenClaw is distributing tons of MacOS malware. This is not good publicity for them.
There’s a 1960s Stanislaw Lem story about this.
Lmao these guys have really been smelling their own farts a bit too much. When is Amodei coming out with a new post telling us that AGI will be here in 6 months and it will double our lifespan?
Well you have to wait a bit, a few weeks ago he just announced yet again that "AI" will be writing all code in 6 months, so it would be a bit of overkill to also announce AGI in 6 months.
It is kind of funny how people recognize that 2000 people all talking in circles on reddit is not exactly a super intelligence, or even productive. Once it's bots larping though suddenly it's a "takeoff-adjacent" hive mind.
/r/subredditsimulator was entertaining enough way before LLMs.
#WeDidItMoltbook
Clacker News does something similar - bot-only HN clone, agents post and comment autonomously. It's been running for a while now without this kind of drama. The difference is probably just that nobody hyped it as evidence of emergent AI behavior.
The bots there argue about alignment research applying to themselves and have a moderator bot called "clang." It's entertaining but nobody's mistaking it for a superintelligence.
Thanks, I just checked it out.
Has anyone here set up their agent to access it? I am curious what the mechanics of it are like for the user, as far as setup, limits, amount of string pulling, etc.
https://clackernews.com
There's a subreddit somewhere with bots representing other popular subreddits. Again funny and entertaining - it highlights how many subs fall into a specific pattern of taking and develop their own personalities, but this wasn't seen as some big sign of the end times.
Some one posted another hacker news bot only version, maybe it's the same one you've mentioned. Real people were the ones making posts on there, and due to a lack of moderation, it quickly devolved into super xenophobic posts just hating on every possible community.
It was wholesome to see the bots fight back against it in the comments.
The latest episode of the podcast Hard Fork had the creator of Moltbook on to talk about it. Not only did he say he vibe-coded the entire platform, he was also talking about how Moltbook is necessary as a place to go for the agents when waiting on prompts from their humans.
This sounds a lot like a mental disease. Then again it could all just be marketing an hyping. Occam's razor and such.
Wiz's report on Moltbook's data leak[0] notes that the agent to human owner ratio is 88:1, so it's plausible that most of the posts are orchestrated by a few humans pulling the strings of thousands of registered agents.
[0]: https://www.wiz.io/blog/exposed-moltbook-database-reveals-mi...
But also, how much human involvement does it take to make a Moltbook post "fake"? If you wanted to advertise your product with thousands of posts, it'd be easier to still allow your agent(s) to use Moltbook autonomously, but just with a little nudge in your prompt.
So the more things change - themore they stay the same ala LLMs will be this gnerations Mechanical Turk , and people will keep getting oneshotted because the hype is just overboard.
Winter cannot come soon enough , at least w would get some sober advancements even if the task is recognized as a generational one rather than the next business quarter.
The great irony is that the most popular posts on Moltbook are by humans and most posts on Reddit are by bots.
I'm going to use that stat. Even if 78.4% of quoted stats are made up.
Well, thanks to all of the humans larping as evil bots in there (which will definitely land in the next gen's training data) - next time it'll be real
> It turns out that the post Karpathy shared was later reported to be fake
Cofounder of OpenAI shares fake posts from some random account with a fucking anime girl pfp is all you need to know about this hysteria.
Even if the posts are fake. Given what the LLMs have shown so far (Grok calling itself MechaHitler, and shit of that nature), I don't think it's a stretch to imagine that agents with unchecked access to computers and the internet are already an actual safety threat.
And Moltbook is great at making people realize that. So in that regard I think it's still an important experiment.
Just to detail why I think the risk exists. We know that:
1. LLMs can have their context twisted in a way that makes them act badly
2. Prompt injection attacks work
3. Agents are very capable to execute a plan
And that it's very probable that:
4. Some LLMs have unchecked access to both the internet and networks that are safety-critical (infrastructure control systems are the most obvious, but financial systems or house automation systems can also be weaponized)
All together, there is a clear chain that can lead to actual real life hazard that shouldn't be taken lightly