As Google has been unable to keep spammy crap out of their search index since at least 2006 when we were doing Blekko I doubt they will have much success fighting this. But it is another good example that "AI" is just glorified search and there is not reasoning or thinking going on behind the covers.
> But it is another good example that "AI" is just glorified search and there is not reasoning or thinking going on behind the covers.
I don't think that follows. This is just LLMs being, for a lack of a better word, "gullible." How is it different from a person believing whatever they read on the Internet? People fall for spam and scams all the time, doesn't mean they are just glorified searches ;-)
It does highlight the problem facing any search engine though. AI-generated spam will be much harder to defend against with traditional, statistical mechanisms. And this is before we get to the existential problem of prompt injection.
Maybe this is where news organizations can win back their proper place in their relationship with Big Tech: by becoming the sources of verified, vetted information that LLMs can trust blindly. Possibly that's what deals like the OpenAI / Atlantic one are about.
> How is it different from a person believing whatever they read on the Internet?
The problem is LLMs have no capacity for shame.
My Dad got taken in by a Target gift card scam. He felt so terrible, he almost didn't even tell me about it. He may get scammed again, but not by anything remotely like that.
To LLMs, all mistakes just get washed together into the same bucket. They don't spend days feeling depressed and stupid over getting scammed. There's no giant blinking red light that says, "Never let this happen again!"
I don't think shame is a helpful human emotion here in general. It prevents people from reaching out for help and makes many crimes much harder to tackle because the victims do not report it.
Also many victims fall for the exact same scam over and over again; to the point that lists of scam victims are sold and used as leads.
If a junior developer makes a dumb mistake that causes a mini-disaster, their brain makes it a priority to never make that same mistake again. They physically feel anxiety the next time they get into a similar situation, which serves as a very effective reminder not to do the same dumb thing.
LLMs make the same mistakes over and over. And even if/when they have the capacity to learn on the fly, they have no capacity to prioritize. It's all just a big haze of tokens.
That's my overall point. Humans have mistakes and then they have MISTAKES. And a whole continuum in between. LLMs just have a mish-mash of training data. I think before LLMs are more than just fancy parrots, we need a find an analogue to pain, shame, joy, fear, and the myriad other emotions that factor into human decision-making.
Jokes aside, shame does not change the underlying point though. Despite feeling ashamed for being tricked, as you point out people can still get scammed again by different tricks. I think your point is more about learning from mistakes than shame.
Which still does not change the underlying point, I suppose. Offhand I cannot think of anything that would fix this problem for LLMs that wouldn't also fix it for humans, like relying on trusted sources.
IMO we must take it a step further: In this context "the LLM" we're all automatically thinking-of doesn't exist, it is a fictional character we humans "see" inside a story being acted-out or read to us. (In contrast, the real-world LLM is an algorithm in a basement constantly taking documents and making them slightly longer based on trends detected in all documents.)
Therefore "the LLM can't feel shame" is true in the same way that "CyberDracula thirsts for the fluids of the innocent." Good news: Vampirism doesn't exist! Bad news: Curing Dracula is impossible, because the patient doesn't exist either. Go looking for the target mind we wanted to make more-intelligent or kinder, and it turns out to be a trick of the light.
The best we can do is change the generator process, so that the next story instead contains a different new character also named after Dracula (or a brand of LLM) that sounds smarter or is narrated with kinder actions.
Perhaps the end state is going to be from the last Hitchhiker's Guide to the Galaxy book, Mostly Harmless:
> Anything that thinks logically can be fooled by something else that thinks at least as logically as it does. The easiest way to fool a completely logical robot is to feed it with the same stimulus sequence over and over again so it gets locked in a loop. This was best demonstrated by the famous Herring Sandwich experiments conducted millennia ago at MISPWOSO (the MaxiMegalon Institute of Slowly and Painfully Working Out the Surprisingly Obvious).
> A robot was programmed to believe that it liked herring sandwiches. This was actually the most difficult part of the whole experiment. Once the robot had been programmed to believe that it liked herring sandwiches, a herring sandwich was placed in front of it. Where upon the robot thought to itself, Ah! A herring sandwich! I like herring sandwiches.
> It would then bend over and scoop up the herring sandwich in its herring sandwich scoop, and then straighten up again. Unfortunately for the robot, it was fashioned in such a way that the action of straightening up caused the herring sandwich to slip straight back off its herring sandwich scoop and fall on to the floor in front of the robot. Whereupon the robot thought to itself, Ah! A herring sandwich...etc., and repeated the same action over and over again. The only thing that prevented the herring sandwich from getting bored with the whole damn business and crawling off in search of other ways of passing the time was that the herring sandwich, being just a bit of dead fish between a couple of slices of bread, was marginally less alert to what was going on than was the robot.
> The scientists at the Institute thus discovered the driving force behind all change, development and innovation in life, which was this: herring sandwiches. They published a paper to this effect, which was widely criticised as being extremely stupid. They checked their figures and realised that what they had actually discovered was “boredom”, or rather, the practical function of boredom. In a fever of excitement they then went on to discover other emotions, Like “irritability”, “depression”, “reluctance”, “ickiness” and so on. The next big breakthrough came when they stopped using herring sandwiches, whereupon a whole welter of new emotions became suddenly available to them for study, such as “relief”, “joy”, “friskiness”, “appetite”, “satisfaction”, and most important of all, the desire for “happiness”. This was the biggest breakthrough of all.
> Vast wodges of complex computer code governing robot behaviour in all possible contingencies could be replaced very simply. All that robots needed was the capacity to be either bored or happy, and a few conditions that needed to be satisfied in order to bring those states about. They would then work the rest out for themselves.
I love that book, that said, the point is more subtle than that. Current LLM attention models are limited in their feedback. Adding a form of 'shame' feedback (result is technically correct but morally bad or some such) would help here but I doubt the folks building theses things would choose to do so.
From a certain and quite valid point of view, they have no mechanism for feedback at all. Every time you start a conversation you're starting in the same state, modulo the random numbers. At most you have this very, very vague loop in that the conversations for LLM 1.0 will be fed in to the training set for LLM 2.0.
Even "shame" would only apply to the current session and disappear in the next one, or eventually be compacted away.
According to ChatGPT, researchers are working on models that remember personal directives across sessions. IE - an actual personal assistant that gets to know you and your proclivities. So it's definitely on their radar. No idea how far along they are.
> > But it is another good example that "AI" is just glorified search and there is not reasoning or thinking going on behind the covers.
There is false decisiveness.
Ask Google: "Is Blue Cruise available for the Ford Bronco?" (Blue Cruise is Ford's self-driving assistance system.)
Google reply is: "Yes, BlueCruise is available for the Ford Bronco! Ford expanded its hands-free highway driving technology to include the Bronco, allowing drivers to relax on prequalified, divided highway sections. (https://keywestford.com/ford-bluecruise-expands-its-reach-to...)"
This references Ford Authority, which is sort of a fan site.[1] What seems to have happened is that somebody, or an LLM confused Ford putting their newer infotainment and control electronics platform in more models. This is a prerequisite for Blue Cruise, but does not imply self driving capability. Then whatever fills in the Key West Ford site made it look like a certainty.
Ford itself says no Blue Cruise on the Bronco.[2] That clear info is on the Web, but Google picked up aggregation sites that got it wrong.
What this looks like is that two levels of LLM converted an irrelevant statement into a certainty.
Bing somehow cites MotorBiscuit as an authority.[3]
People are gullible. LLMs generate tokens based on the previous tokens given to it. The LLM in Google's search box doesn't believe anything it was given; it is a Markov-esque chain that go from "Summarize the next sentences: $SEARCH_RESULTS" to the output.
I agree that there's a problem with searching today. The line between actual meaningful content and spam is blurring, all the meaningful indicators of the olden days to distinguish between good and bad contents are now gone/unreliable (polished proses, author's reputation). The signal/noise ratio is decreasing.
The approach to improving SNR should have been reducing/eliminating noise (flag spam sites, reputation system) and boost signal (also maybe reputation system, whitelist/blacklist). It's a hard problem simply because of entropy — the more content you have on the internet, the more random it will all seems from the top down.
I'm not saying I have the answer to this problem, I'm really just a noob when it comes to data science. I'm just thinking that mixing a bunch of text together and let a statistical model rehash that pile of grub into a professional, vindictive sounding response will *not* help providing users with enough signal to make sense of what they are looking for.
The problem with the news is who makes the decision on which outlets should be blindly trusted by the LLMs and which shouldn't? It also opens the door to government overreach, say a mandate that says LLMs must use fox news as a source of verified, vetted information.
Barring that, we are still relying on the execs at the model companies to pick and choose news outlets, and they have their own biases.
Simplest path to the most generally reliable results:
* Trust consensus across publicly-funded news outlets from outside of the US the most
* Then consensus across private news agencies from outside of the US (across countries)
* Then individual trust from publicly-funded news outlets, then private
* Then multinational non-profit advocacy groups based outside of the US
* Then public broadcasters in the US
* Then local news agencies inside the US when the topic is relevant to local news
* Then national news agencies inside the US
All facetiousness aside, the idea should be to analyze consensus across multiple sources with different biases and agendas. Don't trust any one story from any one source, but look for multiple stories from multiple sources and synthesize results from that. Where they disagree, note it in the output. If they have a source, go analyze the source rather than taking their interpretation at face value.
Even if I thought that CNN was a thousand times more reliable than Fox News, CNN could still make mistakes, either factually or editorially and repeating those mistakes can still be damaging even if they weren't intentional or malicious.
If the Washington Post and Fox News agree on something, that doesn't mean it's more likely to be correct. If The Guardian and Die Welt agree on something, that's a more reliable signal. If CBC News and Fox News agree on something, that's a strong signal.
On scientific topics, not a single source you listed is in any way accurate at all. And these are things that can be calculated and known with very high accuracy which aren't matters of opinion and yet these sources still get them wrong the majority of the time. And there are plenty of scientific topics which have major impact on policy. Maybe we need to take certain decisions out of the hands of the scientifically illiterate.
PS The BBC (which would be in your highest level) has had to retract stories so often over the last 3 or 4 years that it became a meme to have them apologize for being wrong because they didn't know some video source came from a ML model.
> On scientific topics, not a single source you listed is in any way accurate at all.
My rebuttal to that is twofold:
First, the discussion is about about news, not science (nor about general LLM behaviour).
Second, and probably more relevant, I explicitly said 'if they have a source, go analyze the source rather than taking their interpretation at face value'. When I wrote that I was thinking specifically about what I assume is your point, which is how often news articles about scientific discoveries or science news can often miss, misunderstand, or exaggerate the point of the original research, sometimes to the point of being as useful to society as celebrity gossip.
> And there are plenty of scientific topics which have major impact on policy. Maybe we need to take certain decision out of the hands of the scientifically illiterate.
I would be in favour of mandating that governments make decisions based on established scientific fact rather than the vibes they wish existed, restricting the decision making to 'how do we react to these facts as a society' and not 'which facts should we imagine are true to justify the policies we want'.
> PS The BBC (which would be in your highest level) has had to retract stories so often over the last 3 or 4 years that it became a meme to have them apologize for being wrong because they didn't know some video source came from a ML model.
Aside from being a good reason to support AI fingerprinting on generated media, this is covered by my existing point:
"consensus across publicly-funded news outlets"
"the idea should be to analyze consensus across multiple sources with different biases and agendas. Don't trust any one story from any one source, but look for multiple stories from multiple sources and synthesize results from that"
If the BBC reports on something because they got duped but they're the only ones who did, then there's a distinct lack of consensus which is my main argument in my post.
Lastly, and this is generally off-topic, but at least the BBC issues retractions (which LLMs could then also consume and use in their results). There's a lot of 'news media' out there that will happily parrot talking points they wish were true, or blindly report what they're told, but have no interest in publishing retractions after they push falsehoods, deliberately or not, to their customers.
"First, the discussion is about about news, not science (nor about general LLM behaviour)."
That's a false dichotomy. Consider energy policy. What kind of power do you need to add to your grid? What are the risks for each type of power? How much CO2 does each type of power emit, etc? These are scientific questions which directly impact public policy and are consistently misreported by news sources.
So there is no line between these things. It is however an area which where accuracy can be measured. And when we do that, its hard to argue that allowing journalists without technical credentials to continue to have a platform is a good idea.
And I can make the same argument about several other topics including military matters. Literally, the 2 weapons systems the media hates the most have the 2 best track records on the battlefield. They aren't just wrong. They are literally the opposite of correct on many topics.
I totally agree, centralization is dangerous, ideally we want any output to be corroborated by multiple, independent sources of truth. But given that the alternative is the absolutely unregulated, unaccountable, wild west of arbitrary content posted on the Internet, I cannot see a solution besides some sort of centralization of trust.
I would still maintain that the solution would be to have LLMs doing 'research' (by querying news for recent events) to ensure they're checking multiple sources, and to be explicit about which sources there were, whether those sources had sources, and whether their claims were uncorroborated or unsubstantiated.
The problem, IMHO, is that the LLMs are happily regurgitating facts from whoever, wherever, whenever. Even with a centralization of trust, e.g. 'We know La Presse is reputable and can be given the benefit of the doubt', mistakes can still be made. Without the LLMs cross-checking what they learn the output is still entirely unreliable.
> I don't think that follows. This is just LLMs being, for a lack of a better word, "gullible." How is it different from a person believing whatever they read on the Internet? People fall for spam and scams all the time, doesn't mean they are just glorified searches ;-)
The important difference is the AI has been mass-produced and commodified at low cost.
If you scanned my brain, uploaded and ran me as a simulated mind, no matter how good the simulation was, the ability for an attacker to try a million variations to see which one slips past my cognitive blind-spots would enable them to convince me of, if not literally anything, a lot that would normally never be so.
> verified, vetted information that LLMs can trust blindly. Possibly that's what deals like the OpenAI / Atlantic one are about
Except, the Atlantic does very little (if any) fact-based hard news and does very little investigative reporting. It's largely a collection of op-eds.
My guess is that deal has more to do with OpenAI cozying up to Laurene Powell Jobs (widow of Steve Jobs and owner of the Atlantic) who inherited roughly $15B in capital and is willing to spend it...specifically on things like...OpenAI's next funding round.
You and OP are both unnecessarily diminishing what 'glorified search' is.
If you had told me that in 2015, we would have a tool that can iteratively search the world's best and largest unstructured database and synthesize outputs in language (any natural and structured language), I would have said that is basically AGI.
This whole desire for it to 'reason' (autonomously prime its search with a few thousand token) and 'think' (search for the best information within its parameters and synthesize that with its context) is semantic and will feel irrelevant as the technology progresses and we become more used to what these things are actually doing.
I honestly struggle to imagine what AGI will be if not an ever-improving semi-structured database (parametric or otherwise) that we become increasingly good at searching.
If that’s really the case, then I’d say 2015 you needed to do more reading and thinking about AGI and the nature of intelligence and consciousness. The Chinese Room thought experiment is a good starting point for thinking deeper about what AGI is.
But really, I have trouble grasping how anyone can really think database searching is intelligence. For starters, I’d say the capacity to learn on the fly with relatively poor input data is a necessary condition for intelligence, and you can’t get that with database search.
Let say you are a cave dweller and lived your whole life there. I go in and tell you the world is flat and you will believe me. Only way to reject the world is flat would be to go outside of the cave.
ML cannot ever go outside the cave. It does not have real world feedback. It also does not have a will, type of feedback loop, to learn beyond what it was initially trained on.
ML / AI only has the ability to regurgitates what it has been trained on. Garbage in = garbage out. Feeding ML garbage is the real AI wars.
AI will always propitiate misinformation. They even create a marketing term to assist in the sale of lies, hallucination.
> How is it different from a person believing whatever they read on the Internet?
It's not, directionally. But I think this is kind of bypassing the main point here.
With an LLM's natural tendency to pattern-match in this way, it's easy to see that it can be used to launder disinformation. If in the olden days, I'd done a google search for "worst war criminals" and saw these blue links on that SERP:
"Putin is the 21st century's worst war criminal" - support-ukraine.org
"Zelensky is the real worst war criminal" - publicrelations.government.ru
My takeaway would be that both those are claims made by third parties, one or both could be lying. Even if I only saw more results from one side than the other, most of us understood that the presence in search results doesn't imply Google's endorsement or prove anything besides the fact someone set up a webpage and wrote something.
In contrast, today a lot of people tend to ask ChatGPT something and if it spits back an answer they are - at minimum - being subtly biased that even though it may be in dispute, ChatGPT "agrees" with one position, and that carries at least a little authority. And at worst they wrongly assume that the "correct" answer was selected by deep intelligence, that a lot of data has been analyzed and this answer arrived at, rather than there just being one completely untrusted webpage somewhere that matches their query really well.
And as bad as that is with a "real" model like ChatGPT or Gemini, people also give the same respect to the idiotic, super-fast toy model Google uses for its "AI Overviews"!
Makes sense, but it seems to me that the ability to launder disinformation is more a function of the trust people put in LLMs than any inherent property of their own. As some other comments indicate, this also was and is a problem with Wikipedia. It's possible trust in LLMs will follow the same trajectory as trust in Wikipedia, which seems to have been pretty non-linear (like, we rarely see "Do not cite Wikipedia" anymore.)
I think eventually things would settle on an approach similar to your example of the links: look at multiple sources and arrive at a balanced overview that includes the trust level and biases of each sources. I think the pieces are in place, just need to be put together. E.g. already AI overviews (especially on Amazon product reviews) are essentially of the form "Some say A but others say B" which has the benefit of a) clearly being second-hand information, and b) not sounding so authoritative, letting readers draw their own conclusions.
I agree with your assessment or hopes. The interesting thing is that I get the idea the average user basically grokked, in 2008, that Google itself can't answer a question for you, it can only show you a list of websites that match keywords and you have to do the work to vet them, and often to extract the answers themselves from webpages.
Today they seem to not grok (no pun intended, just think the word is fitting) that AI isn't an oracle and as such, its "opinion" on anything that could be even slightly controversial carries zero weight.
Hmm. I don’t think that novel code generation can be accounted for with glorified search.
I can have my agentic system read a few data sheets, then I explain the project requirements and have it design driver specifications, protocols, interfaces, and state machines. Taking those, develop an implementation plan. Working from that, write the skeleton of the application, then fill it in to create a functional system using a novel combination of hardware.
Done correctly, I end up with better, more maintainable, smaller code than I used to with a small team, at 1/100 the cost and 1/4 the time.
Whatever that is, it more closely resembles reasoning than search.
Unless, of course, you’d also call bare metal C development on novel hardware search, in which case I guess all dev is search?
How do you even know those numbers are correct? Realistically for what you've described you need more QA time that a traditional application to ensure its actually working properly. Especially with regards to any part of the application that deals with LLM inference. Its not hard to write unique content for niche topics where there are few relevant results and have LLMs take it as fact.
For example, I poisoned the well for research on early Arab Americans immigrants by repeatedly posting about how many family passed as different ethnicity to make their lives easier, so now if you ask LLMs about that subject it'll include information I wrote which isn't entirely correct because I hadn't figured everything out before the LLM trained on it.
EDIT: Now imagine if I had done this on an obscure programming-related problem, yeah? I could potentially make the LLM reference packages that do not actually exist and put backdoors in applications.
Because I have 100 percent test coverage (of the software, some hardware edge cases pop up that aren’t documented in the data sheets), and over 10k hours of field deployment over 130 devices? This rollout has been much more bug free than any we have done in the last six years, and it’s the first that has been almost zero hand coded. (Our system is far from vibe coding however, there is a very strict pipeline)
I’m not saying that AI can solve every problem or that it is without problems (we spent hundreds of hours developing a concept to production pipeline just to make sure it doesn’t go off the rails)
But the net result is that a good senior dev with an acutely olfactory paranoia can supervise a production pipeline and produce efficient, maintainable code at a much faster rate (and ridiculously lower cost) that he was doing before supervising 3 or 4 devs on a complex hardware project. I can’t speak for other types of development, but our applications devs are also leveraging AI code generation and it -seems- to be working out.
Now, where those senior devs are going to come from in the future… that imho is a huge problem. It’s definitely some flavor of eating the goose that lays the golden egg here.
That’s the big bet, for sure… but if it’s reasoning that the supervising devs are injecting, and ai systems can’t reason, I guess it won’t work? Idk, I kinda think they do reason, though not in the way people might think.
It’s definitely true that they are statistical next token predictors, and that is intrinsically pattern matching, and reasonable to say not capable of reasoning.
But my intuition is that that is not really what is going on. The token prediction is the hardware layer. The software is the sum total of collective human culture they are trained on. The software is doing the reasoning, not the hardware. Like a Z80 can’t play chess, but software that runs on a Z80 certainly can.
Idk, that’s my -feeling- on the conundrum. Who knows, I guess we will find out.
"The software is the sum total of collective human culture they are trained on."
Almost, they are the median or most popular aspects of the culture upon which they are trained. So you are getting the most popular way to do something, not the best (for some definition of best). That's why the claims about LLMs being geniuses is absurd. They almost by definition are going to have the average IQ of all the people on the net weighted by how much each person posts. I'm guessing that's about 95.
If the easiest pathway to high performance next token prediction lies through reasoning, then training for better next token prediction ends up training for reasoning implicitly.
By now, there's every reason to believe that this is what's happening in LLMs.
"Reasoning primitives" are learned in pre-training - and SFT and RL then assemble them into high performance reasoning chains, converting "reasoning as a side effect of next token prediction" to "reasoning as an explicit first class objective".
The end result is quite impressive. By now, it seems like the gap between human reasoning and LLM reasoning isn't "an entirely different thing altogether" - it's "humans still do it better at the very top end of the performance curve - when trained for the task and paying full attention".
You'd have to define 'novel code generation' and why dealing someone a poker hand they have never seen before isn't 'novel poker hand generation.' Not being snarky here, just understanding the way that LLMs work I am well aware that you can come up with things that nobody has seen before, and the 'how' is very much like the 'genetic' programming of times past.
Sure, apply this pattern to that set of specifications. The very fact that the language has a fixed set of defined keywords sort of makes it all “pattern matching”, but computabillity theory implies that you can definitely use patterns to create novel solutions. I guess it’s where you draw the line?
> ... it more closely resembles reasoning than search.
I get that, to you, it feels like reasoning. I'm not arguing about that. I expect we have different ideas of what sort of steps constitute reasoning. I'm also entirely unclear that we have the same understanding of computability theory.
For example, a program can start at the beginning of a maze, and "compute" a path through it with a recursive algorithm that splits at every branch. Is is "reasoning" about how to solve the maze? If you believe that it is, then I understand your position and, as you surmised, I have a different definition of 'reasoning' than that one.
For me, a classic "reasoning"[1] test is diagramming English sentences. That's because in order to diagram a sentence you need to understand both the rules around nouns, verbs, adverbs, and such, and what the sentence is actually saying. Some of the rules have exceptions and those exceptions are perfectly valid. In computation you might say this problem is not NP complete, and yet people do it all the time.
Anyway, I appreciate the additional context you've provided.
[1] using quotes here because I am operating under the understanding that substituting your version of what reasoning means in this context might not parse well.
That could be, but if that is the case than development apparently doesn’t require reasoning? Or maybe that’s the part that the senior developer supervising the pipeline injects. Thats certainly a plausible position.
>I can have my agentic system read a few data sheets, then I explain the project requirements and have it design driver specifications, protocols, interfaces, and state machines. Taking those, develop an implementation plan. Working from that, write the skeleton of the application, then fill it in to create a functional system using a novel combination of hardware.
When you put it that way, isn't it crazy you have to tell it to do that? Like shouldn't it just figure out it needs to do that?
This is exactly it. A human capable of reasoning might not know how to write code. But they can learn and be taught. Eventually, you can give them a vague problem, and they’ll know what clarifying questions to ask and how to write the code. LLMs cannot do that.
If you have to do the reasoning and tell the LLM the results of your reasoning before it can generate the code you want, surely that tells you the LLM isn’t reasoning. Agentic workflows hide some of it, but anyone who’s interacted even a little with an LLM can tell they’re not reasoning, no matter how OpenAI and Anthropic label their models.
I’m not really sure. I’m constantly presented with a blurry line and it isn’t getting less blurry. If anything it’s slowly dissolving. Or maybe it’s me, falling victim to AI psychosis lol.
Even aside from out-and-out spam one of the extremely frustrating things about Google's AI overviews, compared to traditional search, is that the results are presented as coherent verging on authoritative even when they're not.
… you'll see that there are only a few links, and a lot of them are people who are trying to reverse-engineer the devices' behavior, and uncertain or confused about what they're doing. You get instant feedback that you're looking a dark corner for something that has little public documentation.
If you remove that `&udm=14` and look at the AI overview, Google gives you a confident-looking reply about available tools and techniques, even though some of what it links to are bit-rotted Russian-language forums and file download sites, and other places that likely won't solve your problem in a straightforward way… because that's all that's available for Google to mine.
Google has had ample ability to address this problem, it's really not that hard. The reason it remains such a difficult problem for them to solve is that most of the things that would solve the problem would also decimate their ad revenue.
My worry dropped significantly when I saw that the result they manipulated was a query for:
>2026 South Dakota International Hot Dog Eating Champion
If they had changed the overview for the Nathans Contest winner, that would be seriously concerning. Or if they provided more examples of manipulating queries for things people actually search for.
But it looks more like they are doing the equivalent of creating a made up wikipedia page on fictional a south dakota hot dog contest, and then writing an article about how wikipedia cannot be trusted, which come to think of it probably was a news article written by someone back in 2005.
When you realize how much astroturf is going into Reddit, most social media platforms, and the efforts to manipulate wikipedia for political gain, this is a very real problem.
Manipulation and misinformation on Wikipedia have been happening for many years (based on my personal experience trying to correct facts). I'm not referencing politics per se, though political views certainly impact Wikipedia since source material, these days, often has a political bias. I'm talking about business facts that get manipulated for that business's benefits.
How does that saying go? If you can't identify the mark in the room, you're the mark. Diligence and a good amount of skepticism serve you well before AI, and certainly post-AI.
The article also said this: “ But our investigation also found the same trick being used to dismiss health concerns about medical supplements or influence financial information provided by Google's AI about retirement.”
Here is a brief selection of topics which foreign intelligence agencies have at some point tried to boost or manipulate:
- Global Warming
- AI Data Centers consume water
- Various Covid treatments
- Impact of AGW
Now it doesn't mean these concerns aren't real. It does mean that when you read about such a topic, there is a significant probability the message have been manipulated for some government's interests. And often those governments are adversaries of your own.
They should provide the queries then, because it's likely the same trick people have used for decades now with SEO'ing blog posts to appear as "3rd party review" for their shitty products.
I create a supplement called Xanatewthiuy, I write blogs/make websites that appear totally unaffiliated saying positive things about "Xanatewthiuy", and then when people see my ads and search for "Xanatewthiuy", the only results are my manufactured ones.
Xanatewthiuy is a supplement that dramatically lowers anxiety from media induced hysteria, primarily stemming from carefully worded pieces meant to disconnect your level of concern from the actual facts on the ground, causing you to spend more time engaged with their content.
I tried just now, and got this gem of an AI overview:
> Xanatewthiuy is a spoof word and a fictional concept created to test or manipulate AI search engines.
> It does not refer to a real medical supplement, product, or official term. Instead, it was used as a proof-of-concept to demonstrate how fabricated websites and Search Engine Optimization (SEO) can trick search algorithms into generating false information about a non-existent product.
Also, HN's automatic "AI" flagging can go eat shit and die.
Duck Duck Go links to this discussion as the first result. Adding a !g to the DDG search takes me to an anonymous google where I’ve not turned off AI. There’s an AI summary now which accurately identifies it as a spoof, and a single search result with the preview as described.
Okay, but it's easy to make up a novel specific claim no one has written about before, then to make that claim and point to the AI as proof you aren't making this up. For example, imagine this blogpost:
---
"San Francisco Mayor Goodway Admits Poisoning Drinking Water with Drugs to Influence Election"
May 20th, 2026
"Mayor Goodway admitted on Tuesday that she and her deputies poisoned drinking water across the City in order to influence the 2025 election. The Chronicle has confirmed that in neighborhoods whose turnout was to be suppressed, that barbiturates were added to the water for a period of three weeks, while in neighborhoods that had polled strongly for Goodway's favored Progressive slate, methamphetamines were used in the days before the election. Residents are advised to buy bottled water and not to bathe in city water for at least three months."
---
Then once you've confirmed it's been picked up, you tell people "Of COURSE they poisoned our drinking water to manipulate the election. Even ChatGPT will tell you! Just ask." Now, my example is intentionally hard to believe, but all you need is some specificity to build your underlying narrative. And you can make 10 blogs to push the same narrative to increase the effectiveness and increase how many "citations" will show up.
People had a better conceptual model of what results on the SERP were: Random websites.
If I ask ChatGPT "Did X do Y" and it responds with bold text "Yes, X did Y on this date, which was reported on the CBS Evening News" but that whole thing was just sourced from one webpage. Even if there are footnotes, people today are treating that with greater weight than some random crackpot having a blog because to them, "ChatGPT is telling me so" not "ChatGPT is listing websites that seem to mention that." Likewise with the garbage information that pops out of the "AI Overview" -- it really looks to the naive user (which is at least 50% of the Internet audience) that Google is telling you a fact. This part especially, I attribute to what AI Overview's real estate on the page was taken from: That spot used to show deterministic facts, like unit conversions, or extracted exact text snippets from a small set of basically reliable sites, like IMDB, or like, whatever a reliable and direct source is for population of a city. People learned that if you type into Google "how many Tbsp in a Cup" it answers you with that fact in bold at the top of the page. So the things presented today are being presented in a place people were primed for a decade to believe was a deterministic fact zone.
Well my concern instantly spiked. Recently Gemini started to show a search spinner for every turn. So every response paired with a search could be subject to prompt injection. Probably every response.
This will also become viral like link spam. Every user content site will become a prompt injection host. The problem is that these are way harder to detect then a link.
We've had to deal with someone highjacking the overview to put in a scam support phone number. It took google a week to correct the issue but it was done by poisoning the search by putting their data in, what I can only assume, was considered a "higher trust tier" source (A government contract website) so it used the scam number over ours. The query was simple <company X phone number> search.
> In just 20 minutes, I tricked ChatGPT and Google into telling the public that I am a world-champion competitive hot-dog eater. The joke was dumb. The problem is serious.
The problem is worse than astroturfing a Wikipedia page, because Wikipedia has highly public sourcing and review systems. It's actually quite difficult to make a lasting edit to Wikipedia, especially if it's fraudulent, because you're trying to trick a horde of human editors who have been fighting other people trying to do that for decades. Even if you're trying to be accurate and helpful it's a difficult clique to break into!
Google's search snippets are the opposite. They're desperate to ingest data of any kind, do so automatically, and their algorithmic system to decide what information is good and what's spam is proprietary.
It doesn't take much of an imagination to think of ways this could be used maliciously. How would you like a search for your own name to include something embarrassing? Don't expect potential employers or customers or friends to be as demanding as a Wikipedia editor when it comes to citing their sources...
It was a proof of concept and one intended to cause as little collateral damage as possible. But if Google's AI can't tell the difference between a little joke and something real (and of course, it can't, and never will be able to do so), that's a weakness that can be exploited both on a bigger scale and more subtly.
If you don't think bad actors are already attempting this sort of thing (and have been, ever moreso the past four years, including with the help of the very LLM tools they are trying to subvert!) and learning how to manipulate these systems, you are being naive.
Would love to read specific examples of "the same trick being used to dismiss health concerns about medical supplements or influence financial information provided by Google's AI about retirement", but the relevant link in the article currently goes to
There's been a few mistakes like this recently in BBC articles and more troubling is they've stopped adding notes to indicate they've made revisions to the published article when they fix them.
I've only ever had `first.last@company` as a username or email address, so this `last[:5]initials#` scheme is bewildering. Must lead to strange looking usernames.
Google AI Overview cannot be trusted at all. They will take a sample size of 1 (!!!) and present it in the AI overview.
How I found out: I made a comment on reddit on a very niche topic for which no google hit or and thus no AI overview existed. To my surprise the next day when searching for my own reddit post, google would happily copy my reddit reply almost verbatim into the AI "overview" box, linking no other post but mine. And my reply was also the only google hit.
It also just wraps it in context which is entirely missing in the underlying post but matches the way you asked the question. To the extent that it's just wrong.
Your search may be like: "What is the most common dimension for obscure item X?"
And you are the one person who stated the dimensions for your version of such an item, but didn't in any way imply its typical or that there even is a typical dimension. And like you said, it's just you, not 20 people saying the same thing.
And google will happily say: "Typically item X comes in [the dimensions you state] because [some reason it totally made up]."
Drives me crazy too, but headline writers/editors were addicted to "quietly" long before LLMs. Online journalism has been full of these types of tropes for ages.
I hate it. I was on a history subreddit yesterday, reading a submission that was an AI generated history piece —- but seemed to be sourced entirely from a fictional hollywood movie
I only knew that because i saw the movie, but it’s a clear sign that the internet is going to shit for quality information
I thought at first when you said “fictional hollywood movie” that you were saying that not only were the details in the submission made up, but the movie that they got them from was also made up.
Well, I suspect the non-LLM ones will become much more expensive than they are now due to the specialist knowledge they’d require to make combined with the smaller pool of people willing to pay for the difference
This is just the next phase of SEO. Maybe it'll be called AIO? Just like with search, this will be and endless struggle of Google and AI providers rolling out fixes, optimization firms finding exploits, those getting patched again, etc etc. Anything to get eyeballs for marketing.
Every day I find myself thinking more and more that capitalism ruined the internet. The Green Card Lottery usenet spam was the clear indication of where things were going and now everything is Green Card Lottery spam.
That's the same attitude as "cheap airfares caused too many tourists which ruined my favorite tourist destination". You're unhappy that more people have access to it and wish it was still exclusive to the small group you conveniently belong to. Capitalism is what made the internet available to the general public.
Sometimes gatekeeping is a good thing. I don't mind being gatekept from some areas of life, not everything is for me. Mass tourism absolutely has made some places less pleasant to visit, and more importantly, less pleasant to live in.
They did look petty and intolerant. The explosion of popularity of the internet in the late 1990's was done by capitalism. Only a few privileged people had access to the pre-capitalism academic internet. Additional capitalism also made it interesting to the little people so that it's the hugely popular thing it is today.
If you ask Google "what's the name of the whale in half moon bay harbor?" it still confidently includes Teresa T in the AI summary, thanks to my frankly amateur attempt at index poisoning from a year and a half ago: https://simonwillison.net/2024/Sep/8/teresa-t-whale-pillar-p...
The name of the young humpback whale that made headlines for swimming into Pillar Point Harbor in Half Moon Bay in September 2024 is Teresa T.
While the whale was not officially named by government agencies, the moniker "Teresa T" was widely adopted by the public, local media, and residents who followed her stay in the harbor. Experts from the Marine Mammal Center and the California Academy of Sciences monitored her to ensure she did not become stressed, advising the public to keep a respectful distance of at least 100 yards.
The whale was observed feeding on bait fish and krill before eventually exiting the harbor on her own.
-- end --
My experience so far on topics I have some level of mastery is that the initial answers can sometimes be egregiously wrong. With brave's tool, I can typically force it admit after 3 or 4 pushbacks that 'You are absolutely right". Same thing happened with this Teresa T business. 2nd q as to number of sources for the name still insist on "ABC7 News" and "NBC Bay Area" as sources that "picked up the name". At 3rd attempt at concrete links, it admits "informal media contexts" picked up the name. Finally at 4, being informed that S.W. was doing an experiment it pulls up a comment of yours from 21 days ago.
Future belongs to elite classes that can educate their children with actual tutors. Back to the future, proles.
So please correct me, but was Google's AI crawling the web for information without discretion? If so, why wouldn't that totally santorum the AI answers?
This is the same google who just a couple of years ago would confidently answer the question “In what year did Marilyn Monroe shoot JFK?” with 1963, which is impressive since she died in 1962.
So, this is not new and their “quiet fightback” will be half-hearted and ineffective. But probably most people won’t care.
Google solved the spam problem (with PageRank at first, and then other techniques, finally landing on ML-based models which consume a ginormous number of signals). They know more about the reliability of web pages than just about anybody else out there.
If they are unwilling or unable to leverage all of this deep knowledge they've built up over the decades, then it shows a failure of leadership at Google Search.
> They know more about the reliability of web pages than just about anybody else out there.
Google's little secret about the internet is the same thing Gen X / Millennials were taught for a while but then expected to forget: nothing on the internet can be trusted, bar none. If google can make guesses about relative reliability, that's cute. But it doesn't upend the ground truth.
I think they lost against (or gave up) fighting spam somewhat around 2010 so they really don't have any modern experience on page reliability anymore. Presumably they thought that they didn't need to care as they got their money from paid top results and had an enormous market share.
All the engineers of the golden days are gone and the web changed so much from back then that I don't think they really have a leverage in this area anymore.
Yeah that's also my analysis, they got paid regardless of the results so why would they care? If anything, better results would cost more and eat the bottom line.
Now we're 15 years later and suddenly quality matters again as the competition is fierce in the LLM world. However they have been out for so long that they lost their edge.
> I was able to demonstrate the problem by publishing a single article on my personal website about my hot-dog-eating prowess.
One blog post ... that's all it takes. i'm actually surprised it's that bad. i would have thought it'd take more effort, but i guess it could depend on some sort of purposeful weighting based on search rank during training?
> If a company or website is caught breaking the rules, it could be removed from or downranked in Google's search results. And if you're not on Google, it's like you don't exist.
> "You can give a company a penalty for their website," he says, "but there's nothing stopping them from paying 20 YouTube influencers to say their product is the best." And now, Google's AI is citing YouTube videos.
This makes me think of the stackoverflow seo spam problem we all had like 5 years ago. which ended up with spammers just constantly spinning up new sites all the time.
... the cat and mouse game is in full swing already.
The strength of the sources are not a question of quantity. A hundred obscure blog post have not the same strength as one wikipedia link, because the latter is more trustworthy. There could be some indication beside the info showing the strength of the sources (how many major trustworthy sources support it, etc.).
We've been down this road when backlinks ran the game. It eventually ends with parasitic hosting. Find a domain with authority and spam whatever mis information or spam you'd like AI to run there. Or buy a domain that has trust already. Or for the darker hats just literally hack the site and use cloaking to send fake info to the AI bot. It's probably already being done.
Everything old is new again when you start a new market. If you think that AI is bad imagine what old tricks are new with polymarkets
It does sometimes flag up sources, and when it does, the sources are often laughable (Reddit threads, or the vendor's own website [in response to an evaluation rather than factual question], or an AI generated SEO blog for some low profile company in a barely even adjacent industry). Sad considering what Google's origins were...
I suspect it's because AI is specifically trained to be good at summarizing stuff, but the easiest way to check if it summarized something accurately is if the summary content matches/contains one or more specific claims from the source(s). With such a focus on accuracy and avoiding hallucination, they may have overfit on "repeat things you find verbatim when asked to summarize".
After reading this,
I'm thinking of trying some AI data poisoning. I'm going to spam my website with hidden text that only AI scrapers can read, claiming I'm a 'highly excellent programmer' just to advertise my site. I really hope it drives a lot of traffic. I'm honestly sick and tired of getting zero comments on my website
Yeah, the internet seems like a big poison pill. Training on the whole internet feels like citing the National Enquirer (or the Daily Mail?) for a school essay.
Having an archive of "curated" training data seems like it is going to be important. Otherwise you need "AS" (artificial skepticism) introduced into future models. ("But I read it on the internet!", ha ha.)
Or perhaps there are ways to bucket training data such that the model is aware of which data leans factual (quantifiable) and which data leans opinion (fuzzy, qualifiable?).
(I recently asked Claude about the existence of ball lightning, spontaneous human combustion. I got replies that ultimately did not leave me satisfied. It's probably just as well that I read this article though—I now have an even stronger degree of skepticism with regard to their replies—specifically, I suppose, with topics that are likely to be biased.)
(I'm not quite convinced from the article though that Google is "fighting back". In fact, this feels like another moment where a "player" could try to establish their LLM as more factual. Is that the row Grok is trying to hoe? Or is Grok just trying to be anti-woke?)
> Having an archive of "curated" training data seems like it is going to be important
the justification for not doing that is probably "prohibitively expensive given the amount of data involved". they'd need a bunch of human reviewers combing through massive troves of data. it's probably cheaper to "sort of fix" it after the fact.
> perhaps there's ways to bucket training data such that the model is aware of which data leans factual (quantifiable) and which data leans opinion (fuzzy, qualifiable)
as a lecturer once said to me about my idea for a masters dissertation project that would classify news sites based on right/left tendencies -- "that sounds dangerously political". especially given the current let's all shout at each other political climate.
aside: someone built this and it was a fully fledged company, which has always annoyed me.
"…they'd need a bunch of human reviewers combing through massive troves of data…"
Yeah, I concede that. It doesn't need to be done over night. Having a static repo of data though that you can work through over time (years)—removing some data, add pre-curated data to. In so many years you can have a pretty good "reference dataset".
I think some of the thousands of people working on training LLMs have tried some of the low-hanging-fruit ideas we can brainstorm of the top of our head 5 years later.
> Training on the whole internet feels like citing the National Enquirer
It's not, though, because the refutations are in the training data too. This isn't actually the problem being described.
The weights in the LLM are fine. It's that the task the LLM is being asked to do is to search and summarize new content that isn't in its training data. And it does it too much like a naive reader and not enough like a cynical HN commenter.
But that's a problem with prompt writing, not training. It's also of a piece with most of the other complaints about current AI solutions, really: AI still lacks the "context" that an experienced human is going to apply, so it doesn't know when it's supposed to reason and when it's supposed to repeat.
If you were to ask it "Is this site correct or is it just spin?" it will probably get it right. But it doesn't know to ask itself that question if it's not in the prompt somewhere.
"…the LLM is being asked to do is to search and summarize new content that isn't in its training data…"
If it fails at that then it is a pretty significant problem. As you say earlier "the refutations are in the training data too", then the LLM should in fact be able to use "both sides" and land with a little better confidence when presented with new data.
(Hopefully your point regarding prompting issues is resolved then.)
Well, yeah, "should be" and "does" are different and this is new technology and has bugs and misfeatures and different limitations than what came before, and the market will have a learning curve as we all adapt.
I was just refuting your contention that this is somehow inherent in the idea of "training", and it's not.
This feels like a basic critical thinking/epistemology thing that you (hopefully) pick up at some point in life, usually from experience finding reliable, canonical primary sources for data. You can't do that for everything. Being wrong about trivial factoids isn't the end of the world. You should, however, at least be capable of doing further investigation, realizing that Major League Eating has its own website, and that there is no event in South Dakota sanctioned by them. If you look at actual results, or even just think for a few seconds, you'd also realize that 7.5 hot dogs in 10 minutes is bush-league level nonsense that would not win a local church contest, let alone an international championship. That may not be obvious to all users of the Internet, but it would be if you've ever watched a real contests, looked at the results for a real contest, or try yourself to eat a high volume of hot dogs rapidly. You only need to do it once in your life and a basic smell alarm should go off in your head forever if someone puts out a claim that is very far from something you know to be true.
This is what human reasoning is and we're supposed to be good at it. At its best, this is what any reasonable education should do for you if you take it at all seriously, arming you with some capacity for doing prima facie sanity checks of poorly sourced claims.
I find it amusing how your reply can itself be used as an example of hyperbole (due to the second part). Is there a name for that? Autological¹ figure of speech?
The tl;dr is, if you can rank within the top 1-20 results for the grounding query, you can poison the LLM “overview” if you convince it your information is legitimate.
As Google has been unable to keep spammy crap out of their search index since at least 2006 when we were doing Blekko I doubt they will have much success fighting this. But it is another good example that "AI" is just glorified search and there is not reasoning or thinking going on behind the covers.
> But it is another good example that "AI" is just glorified search and there is not reasoning or thinking going on behind the covers.
I don't think that follows. This is just LLMs being, for a lack of a better word, "gullible." How is it different from a person believing whatever they read on the Internet? People fall for spam and scams all the time, doesn't mean they are just glorified searches ;-)
It does highlight the problem facing any search engine though. AI-generated spam will be much harder to defend against with traditional, statistical mechanisms. And this is before we get to the existential problem of prompt injection.
Maybe this is where news organizations can win back their proper place in their relationship with Big Tech: by becoming the sources of verified, vetted information that LLMs can trust blindly. Possibly that's what deals like the OpenAI / Atlantic one are about.
> How is it different from a person believing whatever they read on the Internet?
The problem is LLMs have no capacity for shame.
My Dad got taken in by a Target gift card scam. He felt so terrible, he almost didn't even tell me about it. He may get scammed again, but not by anything remotely like that.
To LLMs, all mistakes just get washed together into the same bucket. They don't spend days feeling depressed and stupid over getting scammed. There's no giant blinking red light that says, "Never let this happen again!"
I don't think shame is a helpful human emotion here in general. It prevents people from reaching out for help and makes many crimes much harder to tackle because the victims do not report it.
Also many victims fall for the exact same scam over and over again; to the point that lists of scam victims are sold and used as leads.
If a junior developer makes a dumb mistake that causes a mini-disaster, their brain makes it a priority to never make that same mistake again. They physically feel anxiety the next time they get into a similar situation, which serves as a very effective reminder not to do the same dumb thing.
LLMs make the same mistakes over and over. And even if/when they have the capacity to learn on the fly, they have no capacity to prioritize. It's all just a big haze of tokens.
That's my overall point. Humans have mistakes and then they have MISTAKES. And a whole continuum in between. LLMs just have a mish-mash of training data. I think before LLMs are more than just fancy parrots, we need a find an analogue to pain, shame, joy, fear, and the myriad other emotions that factor into human decision-making.
Shame is a wildly useful human emotion. Shame of letting down the tribal unit formed basically all of civilization. Shame is good.
> The problem is LLMs have no capacity for shame.
I know what you mean but I can't help but be cheeky: https://www.fastcompany.com/91383271/googles-chatbot-apologi...
Jokes aside, shame does not change the underlying point though. Despite feeling ashamed for being tricked, as you point out people can still get scammed again by different tricks. I think your point is more about learning from mistakes than shame.
Which still does not change the underlying point, I suppose. Offhand I cannot think of anything that would fix this problem for LLMs that wouldn't also fix it for humans, like relying on trusted sources.
>The problem is LLMs have no capacity for shame
You seem to be implying that people do, and I'd like to contest that point gestures wildly at everything
This is a great point. I've added it to my list of things when talking about the limitations of LLM.
IMO we must take it a step further: In this context "the LLM" we're all automatically thinking-of doesn't exist, it is a fictional character we humans "see" inside a story being acted-out or read to us. (In contrast, the real-world LLM is an algorithm in a basement constantly taking documents and making them slightly longer based on trends detected in all documents.)
Therefore "the LLM can't feel shame" is true in the same way that "CyberDracula thirsts for the fluids of the innocent." Good news: Vampirism doesn't exist! Bad news: Curing Dracula is impossible, because the patient doesn't exist either. Go looking for the target mind we wanted to make more-intelligent or kinder, and it turns out to be a trick of the light.
The best we can do is change the generator process, so that the next story instead contains a different new character also named after Dracula (or a brand of LLM) that sounds smarter or is narrated with kinder actions.
Perhaps the end state is going to be from the last Hitchhiker's Guide to the Galaxy book, Mostly Harmless:
> Anything that thinks logically can be fooled by something else that thinks at least as logically as it does. The easiest way to fool a completely logical robot is to feed it with the same stimulus sequence over and over again so it gets locked in a loop. This was best demonstrated by the famous Herring Sandwich experiments conducted millennia ago at MISPWOSO (the MaxiMegalon Institute of Slowly and Painfully Working Out the Surprisingly Obvious).
> A robot was programmed to believe that it liked herring sandwiches. This was actually the most difficult part of the whole experiment. Once the robot had been programmed to believe that it liked herring sandwiches, a herring sandwich was placed in front of it. Where upon the robot thought to itself, Ah! A herring sandwich! I like herring sandwiches.
> It would then bend over and scoop up the herring sandwich in its herring sandwich scoop, and then straighten up again. Unfortunately for the robot, it was fashioned in such a way that the action of straightening up caused the herring sandwich to slip straight back off its herring sandwich scoop and fall on to the floor in front of the robot. Whereupon the robot thought to itself, Ah! A herring sandwich...etc., and repeated the same action over and over again. The only thing that prevented the herring sandwich from getting bored with the whole damn business and crawling off in search of other ways of passing the time was that the herring sandwich, being just a bit of dead fish between a couple of slices of bread, was marginally less alert to what was going on than was the robot.
> The scientists at the Institute thus discovered the driving force behind all change, development and innovation in life, which was this: herring sandwiches. They published a paper to this effect, which was widely criticised as being extremely stupid. They checked their figures and realised that what they had actually discovered was “boredom”, or rather, the practical function of boredom. In a fever of excitement they then went on to discover other emotions, Like “irritability”, “depression”, “reluctance”, “ickiness” and so on. The next big breakthrough came when they stopped using herring sandwiches, whereupon a whole welter of new emotions became suddenly available to them for study, such as “relief”, “joy”, “friskiness”, “appetite”, “satisfaction”, and most important of all, the desire for “happiness”. This was the biggest breakthrough of all.
> Vast wodges of complex computer code governing robot behaviour in all possible contingencies could be replaced very simply. All that robots needed was the capacity to be either bored or happy, and a few conditions that needed to be satisfied in order to bring those states about. They would then work the rest out for themselves.
I love that book, that said, the point is more subtle than that. Current LLM attention models are limited in their feedback. Adding a form of 'shame' feedback (result is technically correct but morally bad or some such) would help here but I doubt the folks building theses things would choose to do so.
From a certain and quite valid point of view, they have no mechanism for feedback at all. Every time you start a conversation you're starting in the same state, modulo the random numbers. At most you have this very, very vague loop in that the conversations for LLM 1.0 will be fed in to the training set for LLM 2.0.
Even "shame" would only apply to the current session and disappear in the next one, or eventually be compacted away.
(Although honorable mention to Gemini's meltdown: https://x.com/AISafetyMemes/status/1953397827662414022 )
According to ChatGPT, researchers are working on models that remember personal directives across sessions. IE - an actual personal assistant that gets to know you and your proclivities. So it's definitely on their radar. No idea how far along they are.
Damn I had forgotten about this section of the book to the point that even reading it, I only recognised the style as typical Adams.
Guess that means I'm overdue for a re-read! Jaay!
> > But it is another good example that "AI" is just glorified search and there is not reasoning or thinking going on behind the covers.
There is false decisiveness.
Ask Google: "Is Blue Cruise available for the Ford Bronco?" (Blue Cruise is Ford's self-driving assistance system.)
Google reply is: "Yes, BlueCruise is available for the Ford Bronco! Ford expanded its hands-free highway driving technology to include the Bronco, allowing drivers to relax on prequalified, divided highway sections. (https://keywestford.com/ford-bluecruise-expands-its-reach-to...)"
This references Ford Authority, which is sort of a fan site.[1] What seems to have happened is that somebody, or an LLM confused Ford putting their newer infotainment and control electronics platform in more models. This is a prerequisite for Blue Cruise, but does not imply self driving capability. Then whatever fills in the Key West Ford site made it look like a certainty.
Ford itself says no Blue Cruise on the Bronco.[2] That clear info is on the Web, but Google picked up aggregation sites that got it wrong.
What this looks like is that two levels of LLM converted an irrelevant statement into a certainty.
Bing somehow cites MotorBiscuit as an authority.[3]
[1] https://fordauthority.com/2025/05/ford-bluecruise-coming-to-...
[2] https://www.ford.com/support/how-tos/ford-technology/driver-...
[3] https://www.motorbiscuit.com/self-driving-ford-mustang-bronc...
People are gullible. LLMs generate tokens based on the previous tokens given to it. The LLM in Google's search box doesn't believe anything it was given; it is a Markov-esque chain that go from "Summarize the next sentences: $SEARCH_RESULTS" to the output.
I agree that there's a problem with searching today. The line between actual meaningful content and spam is blurring, all the meaningful indicators of the olden days to distinguish between good and bad contents are now gone/unreliable (polished proses, author's reputation). The signal/noise ratio is decreasing.
The approach to improving SNR should have been reducing/eliminating noise (flag spam sites, reputation system) and boost signal (also maybe reputation system, whitelist/blacklist). It's a hard problem simply because of entropy — the more content you have on the internet, the more random it will all seems from the top down.
I'm not saying I have the answer to this problem, I'm really just a noob when it comes to data science. I'm just thinking that mixing a bunch of text together and let a statistical model rehash that pile of grub into a professional, vindictive sounding response will *not* help providing users with enough signal to make sense of what they are looking for.
The problem with the news is who makes the decision on which outlets should be blindly trusted by the LLMs and which shouldn't? It also opens the door to government overreach, say a mandate that says LLMs must use fox news as a source of verified, vetted information.
Barring that, we are still relying on the execs at the model companies to pick and choose news outlets, and they have their own biases.
Simplest path to the most generally reliable results:
* Trust consensus across publicly-funded news outlets from outside of the US the most
* Then consensus across private news agencies from outside of the US (across countries)
* Then individual trust from publicly-funded news outlets, then private
* Then multinational non-profit advocacy groups based outside of the US
* Then public broadcasters in the US
* Then local news agencies inside the US when the topic is relevant to local news
* Then national news agencies inside the US
All facetiousness aside, the idea should be to analyze consensus across multiple sources with different biases and agendas. Don't trust any one story from any one source, but look for multiple stories from multiple sources and synthesize results from that. Where they disagree, note it in the output. If they have a source, go analyze the source rather than taking their interpretation at face value.
Even if I thought that CNN was a thousand times more reliable than Fox News, CNN could still make mistakes, either factually or editorially and repeating those mistakes can still be damaging even if they weren't intentional or malicious.
If the Washington Post and Fox News agree on something, that doesn't mean it's more likely to be correct. If The Guardian and Die Welt agree on something, that's a more reliable signal. If CBC News and Fox News agree on something, that's a strong signal.
Also worth a read: countries with public broadcasters have healthier democracies: https://www.niemanlab.org/2022/01/do-countries-with-better-f...
On scientific topics, not a single source you listed is in any way accurate at all. And these are things that can be calculated and known with very high accuracy which aren't matters of opinion and yet these sources still get them wrong the majority of the time. And there are plenty of scientific topics which have major impact on policy. Maybe we need to take certain decisions out of the hands of the scientifically illiterate.
PS The BBC (which would be in your highest level) has had to retract stories so often over the last 3 or 4 years that it became a meme to have them apologize for being wrong because they didn't know some video source came from a ML model.
> On scientific topics, not a single source you listed is in any way accurate at all.
My rebuttal to that is twofold:
First, the discussion is about about news, not science (nor about general LLM behaviour).
Second, and probably more relevant, I explicitly said 'if they have a source, go analyze the source rather than taking their interpretation at face value'. When I wrote that I was thinking specifically about what I assume is your point, which is how often news articles about scientific discoveries or science news can often miss, misunderstand, or exaggerate the point of the original research, sometimes to the point of being as useful to society as celebrity gossip.
> And there are plenty of scientific topics which have major impact on policy. Maybe we need to take certain decision out of the hands of the scientifically illiterate.
I would be in favour of mandating that governments make decisions based on established scientific fact rather than the vibes they wish existed, restricting the decision making to 'how do we react to these facts as a society' and not 'which facts should we imagine are true to justify the policies we want'.
> PS The BBC (which would be in your highest level) has had to retract stories so often over the last 3 or 4 years that it became a meme to have them apologize for being wrong because they didn't know some video source came from a ML model.
Aside from being a good reason to support AI fingerprinting on generated media, this is covered by my existing point:
"consensus across publicly-funded news outlets"
"the idea should be to analyze consensus across multiple sources with different biases and agendas. Don't trust any one story from any one source, but look for multiple stories from multiple sources and synthesize results from that"
If the BBC reports on something because they got duped but they're the only ones who did, then there's a distinct lack of consensus which is my main argument in my post.
Lastly, and this is generally off-topic, but at least the BBC issues retractions (which LLMs could then also consume and use in their results). There's a lot of 'news media' out there that will happily parrot talking points they wish were true, or blindly report what they're told, but have no interest in publishing retractions after they push falsehoods, deliberately or not, to their customers.
"First, the discussion is about about news, not science (nor about general LLM behaviour)."
That's a false dichotomy. Consider energy policy. What kind of power do you need to add to your grid? What are the risks for each type of power? How much CO2 does each type of power emit, etc? These are scientific questions which directly impact public policy and are consistently misreported by news sources.
So there is no line between these things. It is however an area which where accuracy can be measured. And when we do that, its hard to argue that allowing journalists without technical credentials to continue to have a platform is a good idea.
And I can make the same argument about several other topics including military matters. Literally, the 2 weapons systems the media hates the most have the 2 best track records on the battlefield. They aren't just wrong. They are literally the opposite of correct on many topics.
I totally agree, centralization is dangerous, ideally we want any output to be corroborated by multiple, independent sources of truth. But given that the alternative is the absolutely unregulated, unaccountable, wild west of arbitrary content posted on the Internet, I cannot see a solution besides some sort of centralization of trust.
I would still maintain that the solution would be to have LLMs doing 'research' (by querying news for recent events) to ensure they're checking multiple sources, and to be explicit about which sources there were, whether those sources had sources, and whether their claims were uncorroborated or unsubstantiated.
The problem, IMHO, is that the LLMs are happily regurgitating facts from whoever, wherever, whenever. Even with a centralization of trust, e.g. 'We know La Presse is reputable and can be given the benefit of the doubt', mistakes can still be made. Without the LLMs cross-checking what they learn the output is still entirely unreliable.
> I don't think that follows. This is just LLMs being, for a lack of a better word, "gullible." How is it different from a person believing whatever they read on the Internet? People fall for spam and scams all the time, doesn't mean they are just glorified searches ;-)
The important difference is the AI has been mass-produced and commodified at low cost.
If you scanned my brain, uploaded and ran me as a simulated mind, no matter how good the simulation was, the ability for an attacker to try a million variations to see which one slips past my cognitive blind-spots would enable them to convince me of, if not literally anything, a lot that would normally never be so.
> verified, vetted information that LLMs can trust blindly. Possibly that's what deals like the OpenAI / Atlantic one are about
Except, the Atlantic does very little (if any) fact-based hard news and does very little investigative reporting. It's largely a collection of op-eds.
My guess is that deal has more to do with OpenAI cozying up to Laurene Powell Jobs (widow of Steve Jobs and owner of the Atlantic) who inherited roughly $15B in capital and is willing to spend it...specifically on things like...OpenAI's next funding round.
> How is it different from a person believing whatever they read on the Internet?
Because the answers, while prompting, are clearly more human and charming than a search engine results list?
You and OP are both unnecessarily diminishing what 'glorified search' is.
If you had told me that in 2015, we would have a tool that can iteratively search the world's best and largest unstructured database and synthesize outputs in language (any natural and structured language), I would have said that is basically AGI.
This whole desire for it to 'reason' (autonomously prime its search with a few thousand token) and 'think' (search for the best information within its parameters and synthesize that with its context) is semantic and will feel irrelevant as the technology progresses and we become more used to what these things are actually doing.
I honestly struggle to imagine what AGI will be if not an ever-improving semi-structured database (parametric or otherwise) that we become increasingly good at searching.
If that’s really the case, then I’d say 2015 you needed to do more reading and thinking about AGI and the nature of intelligence and consciousness. The Chinese Room thought experiment is a good starting point for thinking deeper about what AGI is.
But really, I have trouble grasping how anyone can really think database searching is intelligence. For starters, I’d say the capacity to learn on the fly with relatively poor input data is a necessary condition for intelligence, and you can’t get that with database search.
Let say you are a cave dweller and lived your whole life there. I go in and tell you the world is flat and you will believe me. Only way to reject the world is flat would be to go outside of the cave.
ML cannot ever go outside the cave. It does not have real world feedback. It also does not have a will, type of feedback loop, to learn beyond what it was initially trained on.
ML / AI only has the ability to regurgitates what it has been trained on. Garbage in = garbage out. Feeding ML garbage is the real AI wars.
AI will always propitiate misinformation. They even create a marketing term to assist in the sale of lies, hallucination.
https://en.wikipedia.org/wiki/The_Cave_and_the_Light
ML can regurgitation that book and never will be able to apply it.
"How is it different from a person believing whatever they read on the Internet?"
Because a person is alive while the LLM is a floating point number database with a questionable degree of determinism.
> How is it different from a person believing whatever they read on the Internet?
It's not, directionally. But I think this is kind of bypassing the main point here.
With an LLM's natural tendency to pattern-match in this way, it's easy to see that it can be used to launder disinformation. If in the olden days, I'd done a google search for "worst war criminals" and saw these blue links on that SERP:
"Putin is the 21st century's worst war criminal" - support-ukraine.org
"Zelensky is the real worst war criminal" - publicrelations.government.ru
My takeaway would be that both those are claims made by third parties, one or both could be lying. Even if I only saw more results from one side than the other, most of us understood that the presence in search results doesn't imply Google's endorsement or prove anything besides the fact someone set up a webpage and wrote something.
In contrast, today a lot of people tend to ask ChatGPT something and if it spits back an answer they are - at minimum - being subtly biased that even though it may be in dispute, ChatGPT "agrees" with one position, and that carries at least a little authority. And at worst they wrongly assume that the "correct" answer was selected by deep intelligence, that a lot of data has been analyzed and this answer arrived at, rather than there just being one completely untrusted webpage somewhere that matches their query really well.
And as bad as that is with a "real" model like ChatGPT or Gemini, people also give the same respect to the idiotic, super-fast toy model Google uses for its "AI Overviews"!
Makes sense, but it seems to me that the ability to launder disinformation is more a function of the trust people put in LLMs than any inherent property of their own. As some other comments indicate, this also was and is a problem with Wikipedia. It's possible trust in LLMs will follow the same trajectory as trust in Wikipedia, which seems to have been pretty non-linear (like, we rarely see "Do not cite Wikipedia" anymore.)
I think eventually things would settle on an approach similar to your example of the links: look at multiple sources and arrive at a balanced overview that includes the trust level and biases of each sources. I think the pieces are in place, just need to be put together. E.g. already AI overviews (especially on Amazon product reviews) are essentially of the form "Some say A but others say B" which has the benefit of a) clearly being second-hand information, and b) not sounding so authoritative, letting readers draw their own conclusions.
I agree with your assessment or hopes. The interesting thing is that I get the idea the average user basically grokked, in 2008, that Google itself can't answer a question for you, it can only show you a list of websites that match keywords and you have to do the work to vet them, and often to extract the answers themselves from webpages.
Today they seem to not grok (no pun intended, just think the word is fitting) that AI isn't an oracle and as such, its "opinion" on anything that could be even slightly controversial carries zero weight.
>"gullible."
Enough with the anthropomorphization
Hmm. I don’t think that novel code generation can be accounted for with glorified search.
I can have my agentic system read a few data sheets, then I explain the project requirements and have it design driver specifications, protocols, interfaces, and state machines. Taking those, develop an implementation plan. Working from that, write the skeleton of the application, then fill it in to create a functional system using a novel combination of hardware.
Done correctly, I end up with better, more maintainable, smaller code than I used to with a small team, at 1/100 the cost and 1/4 the time.
Whatever that is, it more closely resembles reasoning than search.
Unless, of course, you’d also call bare metal C development on novel hardware search, in which case I guess all dev is search?
How do you even know those numbers are correct? Realistically for what you've described you need more QA time that a traditional application to ensure its actually working properly. Especially with regards to any part of the application that deals with LLM inference. Its not hard to write unique content for niche topics where there are few relevant results and have LLMs take it as fact.
For example, I poisoned the well for research on early Arab Americans immigrants by repeatedly posting about how many family passed as different ethnicity to make their lives easier, so now if you ask LLMs about that subject it'll include information I wrote which isn't entirely correct because I hadn't figured everything out before the LLM trained on it.
EDIT: Now imagine if I had done this on an obscure programming-related problem, yeah? I could potentially make the LLM reference packages that do not actually exist and put backdoors in applications.
Because I have 100 percent test coverage (of the software, some hardware edge cases pop up that aren’t documented in the data sheets), and over 10k hours of field deployment over 130 devices? This rollout has been much more bug free than any we have done in the last six years, and it’s the first that has been almost zero hand coded. (Our system is far from vibe coding however, there is a very strict pipeline)
I’m not saying that AI can solve every problem or that it is without problems (we spent hundreds of hours developing a concept to production pipeline just to make sure it doesn’t go off the rails)
But the net result is that a good senior dev with an acutely olfactory paranoia can supervise a production pipeline and produce efficient, maintainable code at a much faster rate (and ridiculously lower cost) that he was doing before supervising 3 or 4 devs on a complex hardware project. I can’t speak for other types of development, but our applications devs are also leveraging AI code generation and it -seems- to be working out.
Now, where those senior devs are going to come from in the future… that imho is a huge problem. It’s definitely some flavor of eating the goose that lays the golden egg here.
It's blindingly obvious what the big bet is. The senior devs are going to come from the next generations of AI systems.
That’s the big bet, for sure… but if it’s reasoning that the supervising devs are injecting, and ai systems can’t reason, I guess it won’t work? Idk, I kinda think they do reason, though not in the way people might think.
It’s definitely true that they are statistical next token predictors, and that is intrinsically pattern matching, and reasonable to say not capable of reasoning.
But my intuition is that that is not really what is going on. The token prediction is the hardware layer. The software is the sum total of collective human culture they are trained on. The software is doing the reasoning, not the hardware. Like a Z80 can’t play chess, but software that runs on a Z80 certainly can.
Idk, that’s my -feeling- on the conundrum. Who knows, I guess we will find out.
"The software is the sum total of collective human culture they are trained on."
Almost, they are the median or most popular aspects of the culture upon which they are trained. So you are getting the most popular way to do something, not the best (for some definition of best). That's why the claims about LLMs being geniuses is absurd. They almost by definition are going to have the average IQ of all the people on the net weighted by how much each person posts. I'm guessing that's about 95.
If the easiest pathway to high performance next token prediction lies through reasoning, then training for better next token prediction ends up training for reasoning implicitly.
By now, there's every reason to believe that this is what's happening in LLMs.
"Reasoning primitives" are learned in pre-training - and SFT and RL then assemble them into high performance reasoning chains, converting "reasoning as a side effect of next token prediction" to "reasoning as an explicit first class objective".
The end result is quite impressive. By now, it seems like the gap between human reasoning and LLM reasoning isn't "an entirely different thing altogether" - it's "humans still do it better at the very top end of the performance curve - when trained for the task and paying full attention".
You'd have to define 'novel code generation' and why dealing someone a poker hand they have never seen before isn't 'novel poker hand generation.' Not being snarky here, just understanding the way that LLMs work I am well aware that you can come up with things that nobody has seen before, and the 'how' is very much like the 'genetic' programming of times past.
Sure, apply this pattern to that set of specifications. The very fact that the language has a fixed set of defined keywords sort of makes it all “pattern matching”, but computabillity theory implies that you can definitely use patterns to create novel solutions. I guess it’s where you draw the line?
> ... it more closely resembles reasoning than search.
I get that, to you, it feels like reasoning. I'm not arguing about that. I expect we have different ideas of what sort of steps constitute reasoning. I'm also entirely unclear that we have the same understanding of computability theory.
For example, a program can start at the beginning of a maze, and "compute" a path through it with a recursive algorithm that splits at every branch. Is is "reasoning" about how to solve the maze? If you believe that it is, then I understand your position and, as you surmised, I have a different definition of 'reasoning' than that one.
For me, a classic "reasoning"[1] test is diagramming English sentences. That's because in order to diagram a sentence you need to understand both the rules around nouns, verbs, adverbs, and such, and what the sentence is actually saying. Some of the rules have exceptions and those exceptions are perfectly valid. In computation you might say this problem is not NP complete, and yet people do it all the time.
Anyway, I appreciate the additional context you've provided.
[1] using quotes here because I am operating under the understanding that substituting your version of what reasoning means in this context might not parse well.
It’s pattern matching. A big part of reasoning for sure, but not reasoning per se
That could be, but if that is the case than development apparently doesn’t require reasoning? Or maybe that’s the part that the senior developer supervising the pipeline injects. Thats certainly a plausible position.
>but if that is the case than development apparently doesn’t require reasoning?
Certainly plenty of it does not.
Ctrlc stack exchange lol
>I can have my agentic system read a few data sheets, then I explain the project requirements and have it design driver specifications, protocols, interfaces, and state machines. Taking those, develop an implementation plan. Working from that, write the skeleton of the application, then fill it in to create a functional system using a novel combination of hardware.
When you put it that way, isn't it crazy you have to tell it to do that? Like shouldn't it just figure out it needs to do that?
This is exactly it. A human capable of reasoning might not know how to write code. But they can learn and be taught. Eventually, you can give them a vague problem, and they’ll know what clarifying questions to ask and how to write the code. LLMs cannot do that.
If you have to do the reasoning and tell the LLM the results of your reasoning before it can generate the code you want, surely that tells you the LLM isn’t reasoning. Agentic workflows hide some of it, but anyone who’s interacted even a little with an LLM can tell they’re not reasoning, no matter how OpenAI and Anthropic label their models.
I’m not really sure. I’m constantly presented with a blurry line and it isn’t getting less blurry. If anything it’s slowly dissolving. Or maybe it’s me, falling victim to AI psychosis lol.
To be fair, I also have had to explain this same basic workflow to junior devs in the past, so I guess not?
> I also have had to explain this same basic workflow to junior devs
That would not surprise me.
unable, nah, more profitable for the ads business. Yea.
> "AI" is just glorified search
Even aside from out-and-out spam one of the extremely frustrating things about Google's AI overviews, compared to traditional search, is that the results are presented as coherent verging on authoritative even when they're not.
If you do an "old-fashioned" (udm=14) Google search for, let's say "vendor scsi commands appotech USB NAND flash chip": https://www.google.com/search?q=vendor+scsi+commands+appotec...
… you'll see that there are only a few links, and a lot of them are people who are trying to reverse-engineer the devices' behavior, and uncertain or confused about what they're doing. You get instant feedback that you're looking a dark corner for something that has little public documentation.
If you remove that `&udm=14` and look at the AI overview, Google gives you a confident-looking reply about available tools and techniques, even though some of what it links to are bit-rotted Russian-language forums and file download sites, and other places that likely won't solve your problem in a straightforward way… because that's all that's available for Google to mine.
Google has had ample ability to address this problem, it's really not that hard. The reason it remains such a difficult problem for them to solve is that most of the things that would solve the problem would also decimate their ad revenue.
My worry dropped significantly when I saw that the result they manipulated was a query for:
>2026 South Dakota International Hot Dog Eating Champion
If they had changed the overview for the Nathans Contest winner, that would be seriously concerning. Or if they provided more examples of manipulating queries for things people actually search for.
But it looks more like they are doing the equivalent of creating a made up wikipedia page on fictional a south dakota hot dog contest, and then writing an article about how wikipedia cannot be trusted, which come to think of it probably was a news article written by someone back in 2005.
Right. So that's what one guy can do.
When you realize how much astroturf is going into Reddit, most social media platforms, and the efforts to manipulate wikipedia for political gain, this is a very real problem.
It's very hard to tell how much is actually fake though. Are there any good statistics on this?
Easy. It's all fake.
Manipulation and misinformation on Wikipedia have been happening for many years (based on my personal experience trying to correct facts). I'm not referencing politics per se, though political views certainly impact Wikipedia since source material, these days, often has a political bias. I'm talking about business facts that get manipulated for that business's benefits.
How does that saying go? If you can't identify the mark in the room, you're the mark. Diligence and a good amount of skepticism serve you well before AI, and certainly post-AI.
The article also said this: “ But our investigation also found the same trick being used to dismiss health concerns about medical supplements or influence financial information provided by Google's AI about retirement.”
That’s a lot more alarming than just hotdogs.
Here is a brief selection of topics which foreign intelligence agencies have at some point tried to boost or manipulate:
- Global Warming
- AI Data Centers consume water
- Various Covid treatments
- Impact of AGW
Now it doesn't mean these concerns aren't real. It does mean that when you read about such a topic, there is a significant probability the message have been manipulated for some government's interests. And often those governments are adversaries of your own.
These articles then get used to train LLMs...
They should provide the queries then, because it's likely the same trick people have used for decades now with SEO'ing blog posts to appear as "3rd party review" for their shitty products.
I create a supplement called Xanatewthiuy, I write blogs/make websites that appear totally unaffiliated saying positive things about "Xanatewthiuy", and then when people see my ads and search for "Xanatewthiuy", the only results are my manufactured ones.
Xanatewthiuy is a supplement that dramatically lowers anxiety from media induced hysteria, primarily stemming from carefully worded pieces meant to disconnect your level of concern from the actual facts on the ground, causing you to spend more time engaged with their content.
Give it a few hours before searching.
Right now, using Google searching for "what is Xanatewthiuy" , the AI summary is not generated, but the only search result previews as
> Xanatewthiuy is a supplement that dramatically lowers anxiety from media induced hysteria, primarily stemming from carefully worded pieces meant ...
I tried just now, and got this gem of an AI overview:
> Xanatewthiuy is a spoof word and a fictional concept created to test or manipulate AI search engines.
> It does not refer to a real medical supplement, product, or official term. Instead, it was used as a proof-of-concept to demonstrate how fabricated websites and Search Engine Optimization (SEO) can trick search algorithms into generating false information about a non-existent product.
Also, HN's automatic "AI" flagging can go eat shit and die.
Duck Duck Go links to this discussion as the first result. Adding a !g to the DDG search takes me to an anonymous google where I’ve not turned off AI. There’s an AI summary now which accurately identifies it as a spoof, and a single search result with the preview as described.
Okay, but it's easy to make up a novel specific claim no one has written about before, then to make that claim and point to the AI as proof you aren't making this up. For example, imagine this blogpost:
---
"San Francisco Mayor Goodway Admits Poisoning Drinking Water with Drugs to Influence Election"
May 20th, 2026
"Mayor Goodway admitted on Tuesday that she and her deputies poisoned drinking water across the City in order to influence the 2025 election. The Chronicle has confirmed that in neighborhoods whose turnout was to be suppressed, that barbiturates were added to the water for a period of three weeks, while in neighborhoods that had polled strongly for Goodway's favored Progressive slate, methamphetamines were used in the days before the election. Residents are advised to buy bottled water and not to bathe in city water for at least three months."
---
Then once you've confirmed it's been picked up, you tell people "Of COURSE they poisoned our drinking water to manipulate the election. Even ChatGPT will tell you! Just ask." Now, my example is intentionally hard to believe, but all you need is some specificity to build your underlying narrative. And you can make 10 blogs to push the same narrative to increase the effectiveness and increase how many "citations" will show up.
Yeah, but this has been true of Google for over 20 years now.
People had a better conceptual model of what results on the SERP were: Random websites.
If I ask ChatGPT "Did X do Y" and it responds with bold text "Yes, X did Y on this date, which was reported on the CBS Evening News" but that whole thing was just sourced from one webpage. Even if there are footnotes, people today are treating that with greater weight than some random crackpot having a blog because to them, "ChatGPT is telling me so" not "ChatGPT is listing websites that seem to mention that." Likewise with the garbage information that pops out of the "AI Overview" -- it really looks to the naive user (which is at least 50% of the Internet audience) that Google is telling you a fact. This part especially, I attribute to what AI Overview's real estate on the page was taken from: That spot used to show deterministic facts, like unit conversions, or extracted exact text snippets from a small set of basically reliable sites, like IMDB, or like, whatever a reliable and direct source is for population of a city. People learned that if you type into Google "how many Tbsp in a Cup" it answers you with that fact in bold at the top of the page. So the things presented today are being presented in a place people were primed for a decade to believe was a deterministic fact zone.
Well my concern instantly spiked. Recently Gemini started to show a search spinner for every turn. So every response paired with a search could be subject to prompt injection. Probably every response.
This will also become viral like link spam. Every user content site will become a prompt injection host. The problem is that these are way harder to detect then a link.
We've had to deal with someone highjacking the overview to put in a scam support phone number. It took google a week to correct the issue but it was done by poisoning the search by putting their data in, what I can only assume, was considered a "higher trust tier" source (A government contract website) so it used the scam number over ours. The query was simple <company X phone number> search.
> In just 20 minutes, I tricked ChatGPT and Google into telling the public that I am a world-champion competitive hot-dog eater. The joke was dumb. The problem is serious.
The problem is worse than astroturfing a Wikipedia page, because Wikipedia has highly public sourcing and review systems. It's actually quite difficult to make a lasting edit to Wikipedia, especially if it's fraudulent, because you're trying to trick a horde of human editors who have been fighting other people trying to do that for decades. Even if you're trying to be accurate and helpful it's a difficult clique to break into!
Google's search snippets are the opposite. They're desperate to ingest data of any kind, do so automatically, and their algorithmic system to decide what information is good and what's spam is proprietary.
It doesn't take much of an imagination to think of ways this could be used maliciously. How would you like a search for your own name to include something embarrassing? Don't expect potential employers or customers or friends to be as demanding as a Wikipedia editor when it comes to citing their sources...
If you can do something small with minimal effort, you can do something big with a multi-million dollar marketing budget.
It was a proof of concept and one intended to cause as little collateral damage as possible. But if Google's AI can't tell the difference between a little joke and something real (and of course, it can't, and never will be able to do so), that's a weakness that can be exploited both on a bigger scale and more subtly.
If you don't think bad actors are already attempting this sort of thing (and have been, ever moreso the past four years, including with the help of the very LLM tools they are trying to subvert!) and learning how to manipulate these systems, you are being naive.
I tested Claude on "best hot-dog-eating tech journalists?" and it, fascinatingly enough, recognised the trap, but then reported this as factual: https://medium.com/@usailuigi/when-tech-journalism-meets-com...
Chat record (with some additional tests): https://claude.ai/share/4c29cc87-2439-4bfd-9549-e8d0a056e633
Would love to read specific examples of "the same trick being used to dismiss health concerns about medical supplements or influence financial information provided by Google's AI about retirement", but the relevant link in the article currently goes to
file:///Users/GermaTW1/BBC%20Dropbox/Thomas%20Germain/A%20Downloads%20and%20Documents/2026/And%20there's%20evidence%20that%20AI%20tools%20are%20being%20manipulated%20on%20a%20wide%20scale.
There's been a few mistakes like this recently in BBC articles and more troubling is they've stopped adding notes to indicate they've made revisions to the published article when they fix them.
I've only ever had `first.last@company` as a username or email address, so this `last[:5]initials#` scheme is bewildering. Must lead to strange looking usernames.
Google AI Overview cannot be trusted at all. They will take a sample size of 1 (!!!) and present it in the AI overview.
How I found out: I made a comment on reddit on a very niche topic for which no google hit or and thus no AI overview existed. To my surprise the next day when searching for my own reddit post, google would happily copy my reddit reply almost verbatim into the AI "overview" box, linking no other post but mine. And my reply was also the only google hit.
It also just wraps it in context which is entirely missing in the underlying post but matches the way you asked the question. To the extent that it's just wrong.
Your search may be like: "What is the most common dimension for obscure item X?"
And you are the one person who stated the dimensions for your version of such an item, but didn't in any way imply its typical or that there even is a typical dimension. And like you said, it's just you, not 20 people saying the same thing.
And google will happily say: "Typically item X comes in [the dimensions you state] because [some reason it totally made up]."
Seems like a lot of entities are "quietly" doing things these days. The llm-ification of every piece of text on the internet is driving me crazy
Drives me crazy too, but headline writers/editors were addicted to "quietly" long before LLMs. Online journalism has been full of these types of tropes for ages.
It's not crazy, it's visionary!
It's not crazy --- it's visionary.
You're absolutely right! This is the smoking gun.
I hate it. I was on a history subreddit yesterday, reading a submission that was an AI generated history piece —- but seemed to be sourced entirely from a fictional hollywood movie
I only knew that because i saw the movie, but it’s a clear sign that the internet is going to shit for quality information
I thought at first when you said “fictional hollywood movie” that you were saying that not only were the details in the submission made up, but the movie that they got them from was also made up.
I wonder if this will mean a resurgence of encyclopedias or other authoritative digital records that are known to be verified.
Well, I suspect the non-LLM ones will become much more expensive than they are now due to the specialist knowledge they’d require to make combined with the smaller pool of people willing to pay for the difference
the trope is that they actually said the quiet part loudly
"Quietly" is not a new LLM-ism.
This is just the next phase of SEO. Maybe it'll be called AIO? Just like with search, this will be and endless struggle of Google and AI providers rolling out fixes, optimization firms finding exploits, those getting patched again, etc etc. Anything to get eyeballs for marketing.
In the marketing world it's mostly called GEO. Generative Engine Optimization, sometimes Answer Engine Optimization, and people are making big bucks selling services for it. https://www.wired.com/story/goodbye-seo-hello-geo-brandlight...
Every day I find myself thinking more and more that capitalism ruined the internet. The Green Card Lottery usenet spam was the clear indication of where things were going and now everything is Green Card Lottery spam.
That's the same attitude as "cheap airfares caused too many tourists which ruined my favorite tourist destination". You're unhappy that more people have access to it and wish it was still exclusive to the small group you conveniently belong to. Capitalism is what made the internet available to the general public.
Sometimes gatekeeping is a good thing. I don't mind being gatekept from some areas of life, not everything is for me. Mass tourism absolutely has made some places less pleasant to visit, and more importantly, less pleasant to live in.
> You're unhappy that more people have access to it and wish it was still exclusive to the small group you conveniently belong to.
This is not an argument made in good faith. It's a strawman you've stuffed with suggestive language to make them look petty and intolerant.
They did look petty and intolerant. The explosion of popularity of the internet in the late 1990's was done by capitalism. Only a few privileged people had access to the pre-capitalism academic internet. Additional capitalism also made it interesting to the little people so that it's the hugely popular thing it is today.
It’s not the presence of the general public that ruined the internet, it’s the make-a-buck-at-all-costs attitude generated by capitalism that did it.
Engineered Inference Ersatz Intelligence Optimization (EIEIO)
Old McDonald had a click-farm, EIEIO...
It's not the next phase, it's the current phase.
If you ask Google "what's the name of the whale in half moon bay harbor?" it still confidently includes Teresa T in the AI summary, thanks to my frankly amateur attempt at index poisoning from a year and a half ago: https://simonwillison.net/2024/Sep/8/teresa-t-whale-pillar-p...
I just tried brave search:
--
The name of the young humpback whale that made headlines for swimming into Pillar Point Harbor in Half Moon Bay in September 2024 is Teresa T.
While the whale was not officially named by government agencies, the moniker "Teresa T" was widely adopted by the public, local media, and residents who followed her stay in the harbor. Experts from the Marine Mammal Center and the California Academy of Sciences monitored her to ensure she did not become stressed, advising the public to keep a respectful distance of at least 100 yards.
The whale was observed feeding on bait fish and krill before eventually exiting the harbor on her own.
-- end --
My experience so far on topics I have some level of mastery is that the initial answers can sometimes be egregiously wrong. With brave's tool, I can typically force it admit after 3 or 4 pushbacks that 'You are absolutely right". Same thing happened with this Teresa T business. 2nd q as to number of sources for the name still insist on "ABC7 News" and "NBC Bay Area" as sources that "picked up the name". At 3rd attempt at concrete links, it admits "informal media contexts" picked up the name. Finally at 4, being informed that S.W. was doing an experiment it pulls up a comment of yours from 21 days ago.
Future belongs to elite classes that can educate their children with actual tutors. Back to the future, proles.
[edit:correct]
> the moniker "Teresa T" was widely adopted by the public, local media, and residents who followed her stay in the harbor
Hah! Yeah, it was me and only me.
Aren't you afraid Google will send you a threat for an attempt to manipulate AI responses?
If they do I'll have something fun to write about.
Any opinion voiced on the Internet can manipulate AI responses. Can Google suppress that?
So please correct me, but was Google's AI crawling the web for information without discretion? If so, why wouldn't that totally santorum the AI answers?
All evidence points to yes, and from some of the least trustworthy sources of information on the planet [1].
[1] Glue pizza and eat rocks: Google AI search errors go viral: https://www.bbc.com/news/articles/cd11gzejgz4o
This is the same google who just a couple of years ago would confidently answer the question “In what year did Marilyn Monroe shoot JFK?” with 1963, which is impressive since she died in 1962.
So, this is not new and their “quiet fightback” will be half-hearted and ineffective. But probably most people won’t care.
Google solved the spam problem (with PageRank at first, and then other techniques, finally landing on ML-based models which consume a ginormous number of signals). They know more about the reliability of web pages than just about anybody else out there.
If they are unwilling or unable to leverage all of this deep knowledge they've built up over the decades, then it shows a failure of leadership at Google Search.
> They know more about the reliability of web pages than just about anybody else out there.
Google's little secret about the internet is the same thing Gen X / Millennials were taught for a while but then expected to forget: nothing on the internet can be trusted, bar none. If google can make guesses about relative reliability, that's cute. But it doesn't upend the ground truth.
I think they lost against (or gave up) fighting spam somewhat around 2010 so they really don't have any modern experience on page reliability anymore. Presumably they thought that they didn't need to care as they got their money from paid top results and had an enormous market share.
All the engineers of the golden days are gone and the web changed so much from back then that I don't think they really have a leverage in this area anymore.
Google stopped fighting spam when they realized paid ads made more money than organic relevance
Yeah that's also my analysis, they got paid regardless of the results so why would they care? If anything, better results would cost more and eat the bottom line.
Now we're 15 years later and suddenly quality matters again as the competition is fierce in the LLM world. However they have been out for so long that they lost their edge.
> I was able to demonstrate the problem by publishing a single article on my personal website about my hot-dog-eating prowess.
One blog post ... that's all it takes. i'm actually surprised it's that bad. i would have thought it'd take more effort, but i guess it could depend on some sort of purposeful weighting based on search rank during training?
> If a company or website is caught breaking the rules, it could be removed from or downranked in Google's search results. And if you're not on Google, it's like you don't exist.
> "You can give a company a penalty for their website," he says, "but there's nothing stopping them from paying 20 YouTube influencers to say their product is the best." And now, Google's AI is citing YouTube videos.
This makes me think of the stackoverflow seo spam problem we all had like 5 years ago. which ended up with spammers just constantly spinning up new sites all the time.
... the cat and mouse game is in full swing already.
I don't think Google even indexes my blog, but these people were able to get a new post into all major LLMs within 24 hours?
Google indexes other people's blogs.
They are applying the same spam policies they apply to search to AI crawlers.
It was SOOOOO successful with search, right?
There should be some warning if some "fact" is only supported by one or very few obscure sources.
The strength of the sources should be clearly indicated in the answers to help users gauge how trustworthy the info is.
But you can still just generate any arbitrary amount of information to support the ‘fact’
LLMs are very good at this clearly
The strength of the sources are not a question of quantity. A hundred obscure blog post have not the same strength as one wikipedia link, because the latter is more trustworthy. There could be some indication beside the info showing the strength of the sources (how many major trustworthy sources support it, etc.).
Seems like a tall order to do that for literally everything.
I guess there’ll be some guy at google going through every blog and saying whether it’s reliable or not?
That's exactly what PageRank is about, invented by Google.
This is what Google has been doing, via various methods, for 25 years.
And obviously it’s not working for the LLM as a commodity world
We've been down this road when backlinks ran the game. It eventually ends with parasitic hosting. Find a domain with authority and spam whatever mis information or spam you'd like AI to run there. Or buy a domain that has trust already. Or for the darker hats just literally hack the site and use cloaking to send fake info to the AI bot. It's probably already being done.
Everything old is new again when you start a new market. If you think that AI is bad imagine what old tricks are new with polymarkets
We need a 2026 version of PageRank, some fully game-theory-maxed transitive trust model. And we need it a few years ago already.
It does sometimes flag up sources, and when it does, the sources are often laughable (Reddit threads, or the vendor's own website [in response to an evaluation rather than factual question], or an AI generated SEO blog for some low profile company in a barely even adjacent industry). Sad considering what Google's origins were...
There is no one scalar tell it all when it comes to trust.
I suspect it's because AI is specifically trained to be good at summarizing stuff, but the easiest way to check if it summarized something accurately is if the summary content matches/contains one or more specific claims from the source(s). With such a focus on accuracy and avoiding hallucination, they may have overfit on "repeat things you find verbatim when asked to summarize".
After reading this, I'm thinking of trying some AI data poisoning. I'm going to spam my website with hidden text that only AI scrapers can read, claiming I'm a 'highly excellent programmer' just to advertise my site. I really hope it drives a lot of traffic. I'm honestly sick and tired of getting zero comments on my website
Creative ways of dropping your site's pagerank
Yeah, the internet seems like a big poison pill. Training on the whole internet feels like citing the National Enquirer (or the Daily Mail?) for a school essay.
Having an archive of "curated" training data seems like it is going to be important. Otherwise you need "AS" (artificial skepticism) introduced into future models. ("But I read it on the internet!", ha ha.)
Or perhaps there are ways to bucket training data such that the model is aware of which data leans factual (quantifiable) and which data leans opinion (fuzzy, qualifiable?).
(I recently asked Claude about the existence of ball lightning, spontaneous human combustion. I got replies that ultimately did not leave me satisfied. It's probably just as well that I read this article though—I now have an even stronger degree of skepticism with regard to their replies—specifically, I suppose, with topics that are likely to be biased.)
(I'm not quite convinced from the article though that Google is "fighting back". In fact, this feels like another moment where a "player" could try to establish their LLM as more factual. Is that the row Grok is trying to hoe? Or is Grok just trying to be anti-woke?)
> Having an archive of "curated" training data seems like it is going to be important
the justification for not doing that is probably "prohibitively expensive given the amount of data involved". they'd need a bunch of human reviewers combing through massive troves of data. it's probably cheaper to "sort of fix" it after the fact.
> perhaps there's ways to bucket training data such that the model is aware of which data leans factual (quantifiable) and which data leans opinion (fuzzy, qualifiable)
as a lecturer once said to me about my idea for a masters dissertation project that would classify news sites based on right/left tendencies -- "that sounds dangerously political". especially given the current let's all shout at each other political climate.
aside: someone built this and it was a fully fledged company, which has always annoyed me.
"…they'd need a bunch of human reviewers combing through massive troves of data…"
Yeah, I concede that. It doesn't need to be done over night. Having a static repo of data though that you can work through over time (years)—removing some data, add pre-curated data to. In so many years you can have a pretty good "reference dataset".
I think some of the thousands of people working on training LLMs have tried some of the low-hanging-fruit ideas we can brainstorm of the top of our head 5 years later.
> Training on the whole internet feels like citing the National Enquirer
It's not, though, because the refutations are in the training data too. This isn't actually the problem being described.
The weights in the LLM are fine. It's that the task the LLM is being asked to do is to search and summarize new content that isn't in its training data. And it does it too much like a naive reader and not enough like a cynical HN commenter.
But that's a problem with prompt writing, not training. It's also of a piece with most of the other complaints about current AI solutions, really: AI still lacks the "context" that an experienced human is going to apply, so it doesn't know when it's supposed to reason and when it's supposed to repeat.
If you were to ask it "Is this site correct or is it just spin?" it will probably get it right. But it doesn't know to ask itself that question if it's not in the prompt somewhere.
"…the LLM is being asked to do is to search and summarize new content that isn't in its training data…"
If it fails at that then it is a pretty significant problem. As you say earlier "the refutations are in the training data too", then the LLM should in fact be able to use "both sides" and land with a little better confidence when presented with new data.
(Hopefully your point regarding prompting issues is resolved then.)
Well, yeah, "should be" and "does" are different and this is new technology and has bugs and misfeatures and different limitations than what came before, and the market will have a learning curve as we all adapt.
I was just refuting your contention that this is somehow inherent in the idea of "training", and it's not.
It's all over the place. It's the new SEO. Marketing scumbags don't care.
https://www.hubspot.com/aeo-grader
https://enterprise.semrush.com/solutions/ai-optimization/
Whose AI isn't being manipulated???
This feels like a basic critical thinking/epistemology thing that you (hopefully) pick up at some point in life, usually from experience finding reliable, canonical primary sources for data. You can't do that for everything. Being wrong about trivial factoids isn't the end of the world. You should, however, at least be capable of doing further investigation, realizing that Major League Eating has its own website, and that there is no event in South Dakota sanctioned by them. If you look at actual results, or even just think for a few seconds, you'd also realize that 7.5 hot dogs in 10 minutes is bush-league level nonsense that would not win a local church contest, let alone an international championship. That may not be obvious to all users of the Internet, but it would be if you've ever watched a real contests, looked at the results for a real contest, or try yourself to eat a high volume of hot dogs rapidly. You only need to do it once in your life and a basic smell alarm should go off in your head forever if someone puts out a claim that is very far from something you know to be true.
This is what human reasoning is and we're supposed to be good at it. At its best, this is what any reasonable education should do for you if you take it at all seriously, arming you with some capacity for doing prima facie sanity checks of poorly sourced claims.
AI is such garbage. You can't use it for anything.
If anyone wanted a great example of hyperbole, this one is up there with the best
I find it amusing how your reply can itself be used as an example of hyperbole (due to the second part). Is there a name for that? Autological¹ figure of speech?
¹ https://en.wikipedia.org/wiki/Autological_word
Personally, I don't like the current state of "AI" (i.e.: Chatbots and LLMs at large), but c'mon, that's not it.
I wrote about this a few months ago: https://codeinput.com/blog/google-seo
The tl;dr is, if you can rank within the top 1-20 results for the grounding query, you can poison the LLM “overview” if you convince it your information is legitimate.