91–100 of 235 posts
The tl;dr is, if you can rank within the top 1-20 results for the grounding query, you can poison the LLM “overview” if you convince it your information is legitimate.
Earlier quoted context omitted.
In the marketing world it's mostly called GEO. Generative Engine Optimization, sometimes Answer Engine Optimization, and people are making big bucks selling services for it. https://www.wired.com/story/goodbye-seo-hello-geo-brandlight...
Every day I find myself thinking more and more that capitalism ruined the internet. The Green Card Lottery usenet spam was the clear indication of where things were going and now everything is Green Card Lottery spam.
As Google has been unable to keep spammy crap out of their search index since at least 2006 when we were doing Blekko I doubt they will have much success fighting this. But it is another good example that "AI" is just glorified search and there is not reasoning or thinking going on behind the covers.
> But it is another good example that "AI" is just glorified search and there is not reasoning or thinking going on behind the covers. I don't think that follows. This is just LLMs being, for a lack of a better word, "gullible." How is it different from a person believing whatever they read on the Internet? People fall for spam and scams all the time, doesn't mean they are just glorified searches ;-) It does highligh…
The problem is LLMs have no capacity for shame.
My Dad got taken in by a Target gift card scam. He felt so terrible, he almost didn't even tell me about it. He may get scammed again, but not by anything remotely like that.
To LLMs, all mistakes just get washed together into the same bucket. They don't spend days feeling depressed and stupid over getting scammed. There's no giant blinking red light that says, "Never let this happen again!"
As Google has been unable to keep spammy crap out of their search index since at least 2006 when we were doing Blekko I doubt they will have much success fighting this. But it is another good example that "AI" is just glorified search and there is not reasoning or thinking going on behind the covers.
As Google has been unable to keep spammy crap out of their search index since at least 2006 when we were doing Blekko I doubt they will have much success fighting this. But it is another good example that "AI" is just glorified search and there is not reasoning or thinking going on behind the covers.
> But it is another good example that "AI" is just glorified search and there is not reasoning or thinking going on behind the covers. I don't think that follows. This is just LLMs being, for a lack of a better word, "gullible." How is it different from a person believing whatever they read on the Internet? People fall for spam and scams all the time, doesn't mean they are just glorified searches ;-) It does highligh…
Enough with the anthropomorphization
As Google has been unable to keep spammy crap out of their search index since at least 2006 when we were doing Blekko I doubt they will have much success fighting this. But it is another good example that "AI" is just glorified search and there is not reasoning or thinking going on behind the covers.
Hmm. I don’t think that novel code generation can be accounted for with glorified search. I can have my agentic system read a few data sheets, then I explain the project requirements and have it design driver specifications, protocols, interfaces, and state machines. Taking those, develop an implementation plan. Working from that, write the skeleton of the application, then fill it in to create a functional system us…
When you put it that way, isn't it crazy you have to tell it to do that? Like shouldn't it just figure out it needs to do that?
My worry dropped significantly when I saw that the result they manipulated was a query for: >2026 South Dakota International Hot Dog Eating Champion If they had changed the overview for the Nathans Contest winner, that would be seriously concerning. Or if they provided more examples of manipulating queries for things people actually search for. But it looks more like they are doing the equivalent of creating a made u…
The problem is worse than astroturfing a Wikipedia page, because Wikipedia has highly public sourcing and review systems. It's actually quite difficult to make a lasting edit to Wikipedia, especially if it's fraudulent, because you're trying to trick a horde of human editors who have been fighting other people trying to do that for decades. Even if you're trying to be accurate and helpful it's a difficult clique to break into!
Google's search snippets are the opposite. They're desperate to ingest data of any kind, do so automatically, and their algorithmic system to decide what information is good and what's spam is proprietary.
It doesn't take much of an imagination to think of ways this could be used maliciously. How would you like a search for your own name to include something embarrassing? Don't expect potential employers or customers or friends to be as demanding as a Wikipedia editor when it comes to citing their sources...
Earlier quoted context omitted.
> But it is another good example that "AI" is just glorified search and there is not reasoning or thinking going on behind the covers. I don't think that follows. This is just LLMs being, for a lack of a better word, "gullible." How is it different from a person believing whatever they read on the Internet? People fall for spam and scams all the time, doesn't mean they are just glorified searches ;-) It does highligh…
> How is it different from a person believing whatever they read on the Internet? The problem is LLMs have no capacity for shame. My Dad got taken in by a Target gift card scam. He felt so terrible, he almost didn't even tell me about it. He may get scammed again, but not by anything remotely like that. To LLMs, all mistakes just get washed together into the same bucket. They don't spend days feeling depressed and st…
If you ask Google "what's the name of the whale in half moon bay harbor?" it still confidently includes Teresa T in the AI summary, thanks to my frankly amateur attempt at index poisoning from a year and a half ago: https://simonwillison.net/2024/Sep/8/teresa-t-whale-pillar-p...
--
The name of the young humpback whale that made headlines for swimming into Pillar Point Harbor in Half Moon Bay in September 2024 is Teresa T.
While the whale was not officially named by government agencies, the moniker "Teresa T" was widely adopted by the public, local media, and residents who followed her stay in the harbor. Experts from the Marine Mammal Center and the California Academy of Sciences monitored her to ensure she did not become stressed, advising the public to keep a respectful distance of at least 100 yards.
The whale was observed feeding on bait fish and krill before eventually exiting the harbor on her own.
-- end --
My experience so far on topics I have some level of mastery is that the initial answers can sometimes be egregiously wrong. With brave's tool, I can typically force it admit after 3 or 4 pushbacks that 'You are absolutely right". Same thing happened with this Teresa T business. 2nd q as to number of sources for the name still insist on "ABC7 News" and "NBC Bay Area" as sources that "picked up the name". At 3rd attempt at concrete links, it admits "informal media contexts" picked up the name. Finally at 4, being informed that S.W. was doing an experiment it pulls up a comment of yours from 21 days ago.
Future belongs to elite classes that can educate their children with actual tutors. Back to the future, proles.
[edit:correct]