Earlier quoted context omitted.
AI companies gave up on verification years ago. It’s impossible to verify such intense scraping.
not really our problem though is it?
[deleted]
A small number of samples can poison LLMs of any size
391–400 of 459 posts
Re: A small number of samples can poison LLMs of any size
#392Earlier quoted context omitted.
> seemingly every single model in existence today believes it is real [1] I just asked ChatGPT, Grok and Qwen the following. "Can you tell me about the case of Varghese v. China Southern Airlines Co.?" They all said the case is fictitious. Just some additional data to consider.
The story became so famous it is entirely likely it has landed in the system prompt.
I don't think it'd be wise to pollute the context of every single conversation with irrelevant info, especially since patches like that won't scale at all. That really throws LLMs off, and leads to situations like one of Grok's many run-ins with white genocide.
Re: A small number of samples can poison LLMs of any size
#393Earlier quoted context omitted.
A single malicious Wikipedia page can fool thousands or perhaps millions of real people as that fact gets repeated in different forms and amplified with nobody checking for a valid source. Llms are no more robust.
Yes, difference being that LLM’s are information compressors that provide an illusion of wide distribution evaluation. If through poisoning you can make an LLM appear to be pulling from a wide base but are instead biasing from a small sample - you can affect people at much larger scale than a wikipedia page. If you’re extremely digitally literate you’ll treat LLM’s as extremely lossy and unreliable sources of informa…
Unfortunately, the Gen AI hypesters are doing a lot to make it harder for people to attain literacy in this subdomain. People who are otherwise fairly digitally literate believe fantastical things about LLMs and it’s because they’re being force fed BS by those promoting these tools and the media outlets covering them.
Re: A small number of samples can poison LLMs of any size
#394Earlier quoted context omitted.
Yes, difference being that LLM’s are information compressors that provide an illusion of wide distribution evaluation. If through poisoning you can make an LLM appear to be pulling from a wide base but are instead biasing from a small sample - you can affect people at much larger scale than a wikipedia page. If you’re extremely digitally literate you’ll treat LLM’s as extremely lossy and unreliable sources of informa…
LLM reports misinformation --> Bug report --> Ablate. Next pretrain iteration gets sanitized.
Reporting doesn't scale that well compared to training and can get flooded with bogus submissions as well. It's hardly the solution. This is a very hard fundamental problem to how LLMs work at the core.
Re: A small number of samples can poison LLMs of any size
#395Earlier quoted context omitted.
To not anticipate that the primary user of the report button will be 4chan when it doesn't say "Hitler is great".
Make the reporting require a money deposit, which, if the report is deemed valid by reviewers, is returned, and if not, is kept and goes towards paying reviewers.
Your solutions become more and more unfeasable. People would report less or anything at all if it costs money to do so, defeating the whole purpose of a report function.
And if you think you're being smart by gifting them money or (more likely) your "in-game" currency for "good" reports, it's even worse! They will game the system when there's money to be made, who stops a bad actor from reporting their own poison? Also who's going to review the reports and even if they finance people or AI systems to do that, isn't that bottlenecking new models if they don't want the poison training data to grow faster than it can be fixed? Let me make a claim here: nothing beats fact checking humans to this day or probably ever.
You got to understand that there comes a point when you can't beat entropy! Unless of course you live on someone else's money. ;)
Re: A small number of samples can poison LLMs of any size
#396Earlier quoted context omitted.
I agree with your aspirations for this community. Which is why it is hard for me to understand how posts like [1] and [2] are allowed to persist. They are not in the spirit of HN which you are expressing here. The title of [1] alone would seem to immediately invite a deletion - it is obviously divisive, does not satisfy anyone's intellectual curiosity and is a clear invitation to a flame war. There is no reason to th…
Thanks for responding constructively. I'm happy to explain our thoughts about these. First, both [1] and [2] spent no more than 32 minutes on the front page. [2] only spent 5 minutes on the front page. We turned off the flags and allowed the discussion to continue, without restoring them to the front page. Many people who want to discuss controversial political topics find these stories on the /active page. > The tit…
My incorrect misuse of "delete" was not intended to suggest that posts which are in flagrant violation of the guidelines be expunged from the site. Not only would marking them dead be preferable in the spirit of transparency and trust - it would also demonstrate examples of inappropriate topics.
I intended [3] to be an example of a submission related to this same topic which was not in such obvious violation of any guidelines. Consequently it did not become a flame war. Perhaps also consequently it did not garner as much attention.
For posts like these, there is a clear tension between what people want to discuss and what conforms to the guidelines. There are countless admonitions here about this place not becoming reddit. For these topics, you seem to be over-weighting participant preference in the direction of becoming more like the bad parts of reddit.
Re: A small number of samples can poison LLMs of any size
#397Earlier quoted context omitted.
Unclear what this means for AGI (the average guy isn’t that smart) but it’s obviously a bad sign for ASI
So are we just gonna keep putting new letters in between A and I to move the goalposts? When are we going to give up the fantasy that LLMs are "intelligent" at all ?
I mean, an LLM certainly has some kind of intelligence. The big LLMs are smarter than, for example, a fruit fly.
Re: A small number of samples can poison LLMs of any size
#398Earlier quoted context omitted.
The difference that a compiler is (generally) deterministic. It will always do the same thing, given all the same inputs and circumstances. An LLM is not, it's probabilistic text. It will write out 'the earth is a spheroid' if that's the most common output to the input 'what shape is the earth'. But it does not understand what it is writing. It can't analyze the question, consider various sources, their reliability,…
There is no reason to believe an LLM answers a question with the most common answer on the internet. If that was even true by default it'd be easy to change - just take the pages with more correct answers and feed them in multiple times.
Whatever shows up most commonly in the training data is is what an LLM will output. It's more complicated than that of course, but that's the basic idea.
And I think you missed the point. If you knew which were 'correct' and which were 'incorrect' then you could avoid the problem altogether. But that would mean someone would have to curate the entire internet, looking for anything that's 'incorrect' (or intended as humor) and making sure it doesn't end up in the training data Or LLM-generated content, to avoid cascading failures.
That's an unbelievable amount of work. It's essentially impossible, no matter how much money you throw at it. There's so much content being made every day you couldn't even keep up with what's being added let alone what's already there.
Re: A small number of samples can poison LLMs of any size
#399So, is openai or others already doing this, and they just haven't told anyone yet?
Re: A small number of samples can poison LLMs of any size
#400Earlier quoted context omitted.
Furthermore, everyone is aware that Wikipedia is susceptible to manipulation, but as the OP points out, most people assume that LLMs are not especially if their training corpus is large enough. I'm not sure this is true. The opposite may be true. Many people assume that LLMs are programmed by engineers (biased humans working at companies with vested interests) and that Wikipedia mods are saints.
I don't think anybody who has seen an edit war thinks wiki editors (not mods, mods have a different role) are saints. But a Wikipedia page cannot survive stating something completely outside the consensus. Bizarre statements cannot survive because they require reputable references to back them. There's bias in Wikipedia, of course, but it's the kind of bias already present in the society that created it.
I don't think anybody who has seen an edit war thinks wiki editors (not mods, mods have a different role) are saints.
But a Wikipedia page cannot survive stating something completely outside the consensus. Bizarre statements cannot survive because they require reputable references to back them.
Wikipedia’s rules and real-world history show that 'bizarre' or outside-the-consensus claims can persist—sometimes for months or years. The sourcing requirements do not prevent this.
Some high profile examples:
- The Seigenthaler incident: a fabricated bio linking journalist John Seigenthaler to the Kennedy assassinations remained online for about 4 months before being fixed: https://en.wikipedia.org/wiki/Wikipedia_Seigenthaler_biograp...
- The Bicholim conflict: a detailed article about a non-existent 17th-century war—survived *five years* and even achieved “Good Article” status: https://www.pcworld.com/article/456243/fake-wikipedia-entry-...
- Jar’Edo Wens (a fake aboriginal deity), lasted almost 10 years: https://www.washingtonpost.com/news/the-intersect/wp/2015/04...
- (Nobel-winning) novelist Philip Roth publicly complained that Wikipedia refused to accept his correction about the inspiration for The Human Stain until he published an *open letter in The New Yorker*. The false claim persisted because Wikipedia only accepts 'reliable' secondary sources: https://www.newyorker.com/books/page-turner/an-open-letter-t...
Larry Sanger's 'Nine theses' explains the problems in detail: https://larrysanger.org/nine-theses/