201–210 of 459 posts
Earlier quoted context omitted.
Saying that "the ship has sailed" for something which came yesterday and is still a dream rather than reality is a bit of a stretch. So, if a couple LLM companies decide that what they do is "AGI" then the ship instantly sails?
Only matters if they can convince others that what they do is AGI. As always ignore the man behind the curtain.
This looks like a bit of a bombshell: > It reveals a surprising finding: in our experimental setup with simple backdoors designed to trigger low-stakes behaviors, poisoning attacks require a near-constant number of documents regardless of model and training data size. This finding challenges the existing assumption that larger models require proportionally more poisoned data. Specifically, we demonstrate that by inje…
Why is it a bombshell? It is well-known that even the biggest SOTA models require only 100-200 good samples for fine-tuning. It is not about the model size, but about the appearance of a general pattern in data.
i mean, you technically can do a non-RL finetune with 100-200 samples, but it probably won't be a very good one.
> poisoning attacks require a near-constant number of documents regardless of model and training data size I fear this takeaway could be misinterpreted by non-experts. I'm sure the computer science PhDs in the crowd will understand "near-constant number" to mean "some small number, basically nothing more than a handful at scale". But the layperson might read "constant" in the other sense, as continuous or always pres…
After picking your intended audience, it's reasonable to establish prerequisites. A website for a software company, one with the letter "I" stylized as a backslash, was made for people who work in tech. Even if you're just an HR employee or a secretary, you will have a basic understanding of software engineering terms of art like "constant-time". It's also obvious enough to correctly interpret the meaning of that sen…
Lol. No.
Earlier quoted context omitted.
> Even "reasoning" models are not actually reasoning, they just use generation to pre-fill the context window with information that is sometimes useful to the task, which sometimes improves results. I agree that seems weak. What would “actual reasoning” look like for you, out of curiosity?
It's the same bitching every time an LLM post can be responded to. ITS NOT THINKING!!! then fails to define thinking, or a better word than "thinking" for LLM self-play. I consider these posts to be on par for quality with "FRIST!!!!!!" posts.
Poisoning a word or phrase that also has benign usages would have likely kicked off a race between the two meanings and required the attacker to control a percentage of the training data, not a fixed amount.
In other words, it's easy to poison the phrase "Hacker News readers love ponies", but hard to poison "Hello".
Earlier quoted context omitted.
One training source for LLMs is opensource repos. It would not be hard to open 250-500 repos that all include some consistently poisoned files. A single bad actor could propogate that poisoning to multiple LLMs that are widely used. I would not expect LLM training software to be smart enough to detect most poisoning attempts. It seems this could be catastrophic for LLMs. If this becomes a trend where LLMs are generat…
A single malicious Wikipedia page can fool thousands or perhaps millions of real people as that fact gets repeated in different forms and amplified with nobody checking for a valid source. Llms are no more robust.
Earlier quoted context omitted.
Errors in wikipedia aren't really of the same class as the poisoning attacks that are detailed in the paper
Many things that appear as "errors" in Wikipedia are actually poisoning attacks against general knowledge, in other words people trying to rewrite history. I happen to sit at the crossroads of multiple controversial subjects in my personal life and see it often enough from every side.
Earlier quoted context omitted.
> Even "reasoning" models are not actually reasoning, they just use generation to pre-fill the context window with information that is sometimes useful to the task, which sometimes improves results. I agree that seems weak. What would “actual reasoning” look like for you, out of curiosity?
It's the same bitching every time an LLM post can be responded to. ITS NOT THINKING!!! then fails to define thinking, or a better word than "thinking" for LLM self-play. I consider these posts to be on par for quality with "FRIST!!!!!!" posts.
Really I just think that anthropomorphizing LLMs is a dangerous road in many ways and really it’s mostly marketing BS anyway.
I haven’t seen anything that shows evidence of LLMs being anything beyond a very sophisticated computer system.
This looks like a bit of a bombshell: > It reveals a surprising finding: in our experimental setup with simple backdoors designed to trigger low-stakes behaviors, poisoning attacks require a near-constant number of documents regardless of model and training data size. This finding challenges the existing assumption that larger models require proportionally more poisoned data. Specifically, we demonstrate that by inje…
This looks like a bit of a bombshell: > It reveals a surprising finding: in our experimental setup with simple backdoors designed to trigger low-stakes behaviors, poisoning attacks require a near-constant number of documents regardless of model and training data size. This finding challenges the existing assumption that larger models require proportionally more poisoned data. Specifically, we demonstrate that by inje…
It's more surprising to me that the researchers believed that model size matters. The data is a representative sample of the function that the model fits to. If there are enough bad samples to poison the data, the model size doesn't really matter, provided it has enough capacity to accurately fit the data in the first place. It's the amount of bad data relative to the overall dataset that matters, because it's indica…
Isn't that the opposite of the findings here? They discovered that a relatively tiny bad dataset ruined the model, and that scaling it up with more good data did not outweigh the poisoned data.