21–30 of 141 posts
Why does this person deserve any kind of support? What’s the point of poisoning LLMs? To put some cursory Luddite roadblock that might delay the technology for a couple of months?
You don't need to vandalize Wikipedia to get this kind of thing to work. Back in September 2024 I named a whale "Teresa T" with just a blog entry and a YouTube video caption: https://simonwillison.net/2024/Sep/8/teresa-t-whale-pillar-p... (For a few glorious weeks if you asked any search-enabled LLM, including Google search previews, for the name of the whale in the Half Moon Bay harbor it confidently replied Teresa…
You don't need to vandalize Wikipedia to get this kind of thing to work. Back in September 2024 I named a whale "Teresa T" with just a blog entry and a YouTube video caption: https://simonwillison.net/2024/Sep/8/teresa-t-whale-pillar-p... (For a few glorious weeks if you asked any search-enabled LLM, including Google search previews, for the name of the whale in the Half Moon Bay harbor it confidently replied Teresa…
This is sort of why "brand" matters; it provides a source of trust.
Encyclopedia Britannica used to be that source of 'facts'. Then it became whatever page-rank told you. Eventually SEO optimization ruined that.
News stories are the same thing. For certain groups, they have their 'independent' publication whose reporting they trust.
You don't need to vandalize Wikipedia to get this kind of thing to work. Back in September 2024 I named a whale "Teresa T" with just a blog entry and a YouTube video caption: https://simonwillison.net/2024/Sep/8/teresa-t-whale-pillar-p... (For a few glorious weeks if you asked any search-enabled LLM, including Google search previews, for the name of the whale in the Half Moon Bay harbor it confidently replied Teresa…
(it probably helps that your name & blog carry some weight, vs. some rando writing something on blogspot or wordpress ;) )
This is basically the same problem of products astroturfing reddit, or SEO optimizing google. You want a new X, and so they heavily go after the keywords associated with it. This is sort of why "brand" matters; it provides a source of trust. Encyclopedia Britannica used to be that source of 'facts'. Then it became whatever page-rank told you. Eventually SEO optimization ruined that. News stories are the same thing. F…
So this means that for bad actors it's more efficient to manufacture brand new fake stories instead of trying to distort the real ones. Don't produce fake articles absolving yourself of a crime, instead produce fake articles accusing your opponent of 100 different things. Then people will fact-check the accusations using LLMs, and since all the sources mentioning those accusations are controlled by you, the LLMs will confirm them.
Most of the popular discourse around AI is still at the level of, "Don't trust the AI, trust the sources!" When it gets to the point where even the sources of simple facts are untrustworthy, the average person just trying to learn some trivia about the world is doomed. Doesn't help that AI media literacy is so primitive compared to how intelligent the models are generally. We're in a marginally better place than we w…
Wikipedia calls this https://en.wikipedia.org/wiki/Citogenesis (after XKCD coined it). You can expect the spicy autocomplete to feed you flattering bullshit. It may cite Wikipedia (it shouldn't), but you should go check out those citations, and validate the claims yourself. It's the least you can do. And if the cited source is Wikipedia... check Wikipedia's sources too. Wikipedians try their best to provide you with…
(Norm Macdonald voice) Or so the Germans would have us believe...!