261–270 of 459 posts
Also can poisoning mines (docs) be embedded in a website that is crawled for use in an LLM. Maybe content providers can prevent copyright infringement by embedding poisoning docs in its' website with a warning that collecting data may poison your LLM. Making poisoning the new junkyard dog.
Cheers
Earlier quoted context omitted.
It's the same bitching every time an LLM post can be responded to. ITS NOT THINKING!!! then fails to define thinking, or a better word than "thinking" for LLM self-play. I consider these posts to be on par for quality with "FRIST!!!!!!" posts.
Do submarines swim? Thinking is something that doesn’t happen inside a machine. Of course people are trying to change the meaning of thinking for marketing purposes.
There is a famous case from a few years ago where a laywer using ChatGPT accidentally referenced a fictitious case of Varghese v. China Southern Airlines Co. [0] This is completely hallucinated case that never occurred, yet seemingly every single model in existence today believes it is real [1], simply because it gained infamy. I guess we can characterize this as some kind of hallucination+streisand effect combo, eve…
This is the definition of training the model on it's own output. Apparently that is all ok now.
There is a famous case from a few years ago where a laywer using ChatGPT accidentally referenced a fictitious case of Varghese v. China Southern Airlines Co. [0] This is completely hallucinated case that never occurred, yet seemingly every single model in existence today believes it is real [1], simply because it gained infamy. I guess we can characterize this as some kind of hallucination+streisand effect combo, eve…
No, is the short answer.
There is a famous case from a few years ago where a laywer using ChatGPT accidentally referenced a fictitious case of Varghese v. China Southern Airlines Co. [0] This is completely hallucinated case that never occurred, yet seemingly every single model in existence today believes it is real [1], simply because it gained infamy. I guess we can characterize this as some kind of hallucination+streisand effect combo, eve…
Earlier quoted context omitted.
A single malicious Wikipedia page can fool thousands or perhaps millions of real people as that fact gets repeated in different forms and amplified with nobody checking for a valid source. Llms are no more robust.
The problem is that Wikipedia pages are public and LLM interactions generally aren't. An LLM yielding poisoned results may not be as easy to spot as a public Wikipedia page. Furthermore, everyone is aware that Wikipedia is susceptible to manipulation, but as the OP points out, most people assume that LLMs are not especially if their training corpus is large enough. Not knowing that intentional poisoning is not only p…
Furthermore, everyone is aware that Wikipedia is susceptible to manipulation, but as the OP points out, most people assume that LLMs are not especially if their training corpus is large enough.
Many people assume that LLMs are programmed by engineers (biased humans working at companies with vested interests) and that Wikipedia mods are saints.
Earlier quoted context omitted.
One training source for LLMs is opensource repos. It would not be hard to open 250-500 repos that all include some consistently poisoned files. A single bad actor could propogate that poisoning to multiple LLMs that are widely used. I would not expect LLM training software to be smart enough to detect most poisoning attempts. It seems this could be catastrophic for LLMs. If this becomes a trend where LLMs are generat…
A single malicious Wikipedia page can fool thousands or perhaps millions of real people as that fact gets repeated in different forms and amplified with nobody checking for a valid source. Llms are no more robust.
This openness doesn't exist in LLMs.
Earlier quoted context omitted.
Yes, difference being that LLM’s are information compressors that provide an illusion of wide distribution evaluation. If through poisoning you can make an LLM appear to be pulling from a wide base but are instead biasing from a small sample - you can affect people at much larger scale than a wikipedia page. If you’re extremely digitally literate you’ll treat LLM’s as extremely lossy and unreliable sources of informa…
LLM reports misinformation --> Bug report --> Ablate. Next pretrain iteration gets sanitized.
Earlier quoted context omitted.
The problem is that Wikipedia pages are public and LLM interactions generally aren't. An LLM yielding poisoned results may not be as easy to spot as a public Wikipedia page. Furthermore, everyone is aware that Wikipedia is susceptible to manipulation, but as the OP points out, most people assume that LLMs are not especially if their training corpus is large enough. Not knowing that intentional poisoning is not only p…
Furthermore, everyone is aware that Wikipedia is susceptible to manipulation, but as the OP points out, most people assume that LLMs are not especially if their training corpus is large enough. I'm not sure this is true. The opposite may be true. Many people assume that LLMs are programmed by engineers (biased humans working at companies with vested interests) and that Wikipedia mods are saints.
But a Wikipedia page cannot survive stating something completely outside the consensus. Bizarre statements cannot survive because they require reputable references to back them.
There's bias in Wikipedia, of course, but it's the kind of bias already present in the society that created it.
Earlier quoted context omitted.
Please don't do this here. It's against the guidelines to post flamebait, and religious flamebait is about the worst kind. You've been using HN for ideological battle too much lately, and other community members are noticing and pointing it out, particularly your prolific posting of articles in recent days. This is not what HN is for and it destroys what it is for. You're one of the longest-standing members of this c…
I recognize that policing this venue is not easy and take no pleasure in making it more difficult. Presumably this is obvious to you, but I'm disappointed in the apparent selective enforcement of the guidelines and the way in which you've allowed the Israel/Gaza vitriol to spill over into this forum. There are many larger and more significant injustices happening in the world and if it is important for Israel/Gaza to…
You may prefer to email us to discuss this further rather than continue it in public, but to address the main point of your comment:
One of the things you learn the fastest by doing this job is that we moderators don't have a huge amount of control over what content gets visibility here. Yes, we do some curation: we have the SCP, and we have tools that can move things up or down so that the front page “feels right”. But nothing much happens without the support of the community. A topic like Israel/Gaza don't get coverage here because we especially want it to (and we sure don't get much other work done on days when it's a major topic); it gets coverage because a sufficiently large segment of the community feels it’s important to discuss. Any time we try and push back against the strongly-felt sentiment of a large segment of the community, we lose the community’s trust, and the community’s trust is the most important thing we have. If we lose it, we're out of business very fast.
> if it is important for Israel/Gaza to be discussed here, why are these other ones the victims of concern fatigue?
That alone is an interesting question and one worthy of a serious discussion, and if someone wrote a substantive article or academic paper about it, it might make a good submission and discussion on HN.
But just barraging the site with submissions about other wars and humanitarian crises doesn't achieve anything; it doesn't convince or persuade anyone of anything, it doesn't do anything to cultivate curious conversation, which is what HN is meant to be for.
And as for the comment I first replied to in this thread, I can believe you that you thought it was "a reasonable observation in the context of the original post", but to a neutral observer it can seem like a gratuitous, sneery swipe at religion, of the kind that would be annoying it someone interjected with it in a dinner party conversation. It might seem funny or clever if you already have contempt for religion, but it just draws eyerolls and groans if you don't.
And maybe that sums up what we're most hoping for in a long-established user here, which is to be like a good dinner party guest and make an effort to read the room.