AI Slop vs. OSS Security
devansh.bearblog.dev
AI Slop vs. OSS Security
1–10 of 124 posts
Re: AI Slop vs. OSS Security
#2> The model has no concept of truth—only of plausibility.
This is such an important problem to solve, and it feels soluble. Perhaps a layer with heavily biased weights, trained on carefully curated definitional data. If we could train in a sense of truth - even a small one - many of the hallucinatory patterns disappear.
Hats off to the curl maintainers. You are the xkcd jenga block at the base.
Re: AI Slop vs. OSS Security
#3> The model has no concept of truth—only of plausibility. This is such an important problem to solve, and it feels soluble. Perhaps a layer with heavily biased weights, trained on carefully curated definitional data. If we could train in a sense of truth - even a small one - many of the hallucinatory patterns disappear. Hats off to the curl maintainers. You are the xkcd jenga block at the base.
I am assuming that millions of dollars have already been spent trying to get LLMs to hallucinate less.
Even if Problems feel soluble, they often aren't. You might have to invent an entirely new paradigm of text generation to solve the hallucination problem. Or it could be the Collatz Conjecture of LLMs, that it "feels" so possible, but you never really get there.
Re: AI Slop vs. OSS Security
#4Wealth generated on top of underpaid labor is a reoccurring theme -- and in this case maybe surprisingly exacerbated by LLMs.
Would this be different if the underlying code had a viral license? If google's infrastructure was built on a GPL'ed libcurl [0], would they have investment in the code/a team with resources to evaluate security reports (slop or otherwise)? Ditto for libxml.
Does GPL help the linux kernel get investment from it's corporate users?
[0] Perhaps an impossible hypothetical. Would google have skipped over the imaginary GPL'ed libcurl or libxml for a more permissively licensed library? And even if they didn't, would a big company's involvement in an openly developed ecosystem create asymmetric funding/goals, a la XMPP or Nix?
Re: AI Slop vs. OSS Security
#5Wealth generated on top of underpaid labor is a reoccurring theme -- and in this case maybe surprisingly exacerbated by LLMs. Would this be different if the underlying code had a viral license? If google's infrastructure was built on a GPL'ed libcurl [0], would they have investment in the code/a team with resources to evaluate security reports (slop or otherwise)? Ditto for libxml. Does GPL help the linux kernel get…
Copyleft licenses are made to support freedom for everyone and particularly end-users. They only limit freedom of developers / maintainers to exploit the code and users.
> Does GPL help the linux kernel get investment from it's corporate users?
GPL has helped "linux kernel the project" greatly, but companies invest in it out of their self-interest. They want to benefit from upstream improvements and playing nicely by upstreaming changes is just much cheaper than maintaining own kernel fork.
On other side you have companies like Sony that used BSD OS code for their game consoles for decades and contributed shit.
So... Two unrelated things.
Re: AI Slop vs. OSS Security
#6> Requiring technical evidence such as screencasts showing reproducibility, integration or unit tests demonstrating the fault, or complete reproduction steps with logs and source code makes it much harder to submit slop.
If this isn't already a requirement, I'm not sure I understand what even non-AI-generated reports look like. Isn't the bare-minimum of CVE reporting a minimally reproducible example? Like, even if you find some function, that for example doesn't do bounds-checking on some array, you can trivially write some unit testing code that's able to break it.
Re: AI Slop vs. OSS Security
#7> The model has no concept of truth—only of plausibility. This is such an important problem to solve, and it feels soluble. Perhaps a layer with heavily biased weights, trained on carefully curated definitional data. If we could train in a sense of truth - even a small one - many of the hallucinatory patterns disappear. Hats off to the curl maintainers. You are the xkcd jenga block at the base.
I am assuming that millions of dollars have already been spent trying to get LLMs to hallucinate less. Even if Problems feel soluble, they often aren't. You might have to invent an entirely new paradigm of text generation to solve the hallucination problem. Or it could be the Collatz Conjecture of LLMs, that it "feels" so possible, but you never really get there.
Nuclear fusion was always 30 years away (c)
Re: AI Slop vs. OSS Security
#8The solution isn't to block aggressively or to allow everything, but to prioritize.
Put accounts older than the AI boom at the top, and allow them to give "referrals", ie stake a part of their own credibility to boost another account on the priority ladder.
Referral systems are very efficient at filtering noise.
Re: AI Slop vs. OSS Security
#9> This is the fundamental problem: AI can generate the form of security research without the substance.
I think this is the fundamental problem of LLMs in general. Some of the time looks just enough right to seem legitimate. Luckily the rest of the time it doesn’t.
Re: AI Slop vs. OSS Security
#10> The model has no concept of truth—only of plausibility. This is such an important problem to solve, and it feels soluble. Perhaps a layer with heavily biased weights, trained on carefully curated definitional data. If we could train in a sense of truth - even a small one - many of the hallucinatory patterns disappear. Hats off to the curl maintainers. You are the xkcd jenga block at the base.
The "fact database" is the old AI solution, e.g. Cycorp; it doesn't quite work either. Knowing what is true is a really hard, unsolved problem in philosophy, see e.g. https://en.wikipedia.org/wiki/Gettier_problem . The secret to modern AI is just to skip that and replace unsolvable epistemology with "LGTM", then sell it to investors.