Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SQLite Critical CVEs or LLM Slop?

research.jfrog.com

Post reply on HN
Print view View original

251–260 of 406 posts

Re: SQLite Critical CVEs or LLM Slop?

#251

Post by Jensson »

diydsp wrote: post #246
Jensson wrote: post #215

Earlier quoted context omitted.

The rate of progress can be high and they can also be dumb next token predictors. Not sure why that is hard to understand. These models can do a lot of things but they also can't do a lot of things. In order to use these models effectively you have to understand that they are next token predictors and how that allows it to do what they do.

Much of an LLM's capability comes from the structure encoded in its learned representations. The probabilistic outputs are primarily a way of expressing uncertainty and generating fluent text, while compression during training is what forces the model to discover that underlying structure.

> Much of an LLM's capability comes from the structure encoded in its learned representations

And thats encoded as a set of next token predictions. So the way to see how reliably it solves a problem is to look at the chain of predictions, and see where it is unreliable at finding the next spot, or where it always fails and you need to add that link to the dataset to train it.

This isn't magic, today we understand pretty well how to add new skills to LLM, and the better this is understood the faster progress will be.

This also means that if a context doesn't have any good predictions, it will produce a dumb prediction for that context. This results in these bad outcomes, because currently LLM doesn't have a map for where predictions are good or bad.

Re: SQLite Critical CVEs or LLM Slop?

#252

Post by RevEng »

mwkaufma wrote: post #142
bluGill wrote: post #72

Earlier quoted context omitted.

The real question is will [when will] these vulnerabilities get fixed faster than they can be found. LLMs are finding a lot of real bugs that have existed without being noticed for 20+ years. It is generally expected [hoped] that in a short time all those 20 year old bugs will be fixed and LLMs will settle down to just things introduced in the last month (or perhaps this pull request) and thus be manageable.

Won't an LLM, prompted for a CVE report, always hallucinate one even if it isn't there? Why would they "settle down"?

That's becoming less common. The issue is that, if training only contains examples of CVE reports, then asking for one will get you one, even if it's not grounded in reality. But if training consists of many negative cases with the response, "I can't find one," them that will become the default unless evidence pushes it towards a report. It's a classic class size problem. SOTA models seem to have this largely solved, though a determined person (just like a determined boss) can prompt one into giving a false answer just to satisfy the request.

Re: SQLite Critical CVEs or LLM Slop?

#253

Post by pixl97 »

ymir_e wrote: post #84
bluGill wrote: post #57

Earlier quoted context omitted.

All the big maintainers I know of disagree. Curl for example is getting overwhelmed with real issues that LLMs are finding. A year ago most LLM found issues where false positives (the code is correct), or hallucinations (the code in question doesn't even exist) - but that has changed and now LLMs are finding real issues in code. Many different projects have reported the same rules.

Absolutely. I'm not sure how universal this is, but I saw George Hotz tweeted something about Mythos not being such a big deal since he's able to find security issues without a claimed $20,000 in token spend per vulnerability. The reality is that humans aren't perfect, so we do not make "perfect programs". It is also possible to test it yourself to see how good LLMs are at finding software issues: Find any open sourc…

So we just need a redundant array of George Hotz, I wonder how much that costs?

This is where quantity becomes a quality. Actors with huge budgets and at least a small team to keep the AIs on track now have an immense amount of potential power. Anthropic has somewhat resisted in giving state actors like this full access to 'dangerous' models, but it appears other companies like OpenAI have no such reservations.

Before such states may have difficulty hiring enough security researchers/hackers. Now you can spin them up. Someone with a few data centers of GPU compute and enough proxies could probably blitzkrieg the internet in a day and capture ungodly amounts of information.

Re: SQLite Critical CVEs or LLM Slop?

#254

Post by gbnwl »

unknownfuture wrote: post #239
Yajirobe wrote: post #237

Earlier quoted context omitted.

Tell me how a 'nExT toKeN prEdIcTor' can make breakthroughs in math or play a game of chess. These activities aren't pure symbol manipulation, they require actual understanding at some level.

Eh, I'm not going to litigate your claims. My point is it's silly to whine that HN is a place where multiple points of view on the topic are aired out and discussed. If you want a personal echo chamber where only your own beliefs are affirmed and anything else is flagged off or downvoted, I'm sure you can go find one or, worst case, vibe code one into existence.

Fair so let me be clear. I’m whining because the “next token predictor” reductionist point of view has been wrong and is only growing more wrong with time. Clearly these things can do things that actually matter. Do you disagree?

Re: SQLite Critical CVEs or LLM Slop?

#255

Post by a3mrouter1 »

gortok wrote: post #52

We can chalk this up as another example of over-exhuberance by what folks believe LLMs can accomplish vs. what they actually are. LLM-based “AI” is able to use its vast corpus of inputs and calculate the most statistically likely output in a given situation. It is probabilistic, and when you are dealing with probabilities in a situation where certainties, not probabilities, matter, you’re going to get dinged on credi…

[flagged]

Re: SQLite Critical CVEs or LLM Slop?

#256

Post by estearum »

tadfisher wrote: post #189
estearum wrote: post #162

Earlier quoted context omitted.

> but being able to generate probabilistic outputs based on inputs is not what makes us intelligent. ??? Of course it is. The brain is mechanically not capable of doing anything other than that. Do you believe the brain is something other than a bundle of probabilistic physical interactions? Or are brains not the source of what we call intelligence?

This deserves about as large of a "[citation needed]" as one could draw. Are you well-studied in neuroscience?

Nothing in the universe is anything other than a chain of reactions, amigo.

The brain is an object in the universe.

The universe has quantum behaviors (fully random, not a source of intelligence) and it has deterministic behaviors (fully non-random). Many of those deterministic behaviors are so complex that they're easier to analyze and describe as probabilistic, which is where most brain input → output relationships land.

Please point to any evidence whatsoever that the brain has some third type of interaction going on that has never been observed anywhere in the entire universe, then we can have a discussion about it.

Re: SQLite Critical CVEs or LLM Slop?

#257

Post by Jblx2 »

Sohcahtoa82 wrote: post #210
bwfan123 wrote: post #183

Earlier quoted context omitted.

> Now, we all pay the consequence, to the tune of hundreds of thousands if not millions of dollars of wasted productivity from teams that have to deal with the resulting fall-out of this usage of “AI”. Brandolini's principle in action. It takes 10 times more energy to refute BS than to generate it. A related analogy to computing: it is easy to generate propositions, but hard to test if a given proposition is satisfia…

> Brandolini's principle I much prefer the alternative name: the Bullshit Asymmetry Principle.

Seems like most of it is covered by: Entropy increases.

Re: SQLite Critical CVEs or LLM Slop?

#258

Post by pixl97 »

mwkaufma wrote: post #142
bluGill wrote: post #72

Earlier quoted context omitted.

The real question is will [when will] these vulnerabilities get fixed faster than they can be found. LLMs are finding a lot of real bugs that have existed without being noticed for 20+ years. It is generally expected [hoped] that in a short time all those 20 year old bugs will be fixed and LLMs will settle down to just things introduced in the last month (or perhaps this pull request) and thus be manageable.

Won't an LLM, prompted for a CVE report, always hallucinate one even if it isn't there? Why would they "settle down"?

Hence future CVEs on a lot of projects will likely demand working exploit code for testing. This at least attempts to lock the issue down to something more tangible.

Re: SQLite Critical CVEs or LLM Slop?

#259

Post by Blackthorn »

prh8 wrote: post #172
elmer2 wrote: post #168

Earlier quoted context omitted.

Many people with no skills are taking advantage of the LLM craze to artificially inflate their own value. I see it every day on LinkedIn. People that previously have barely any experience in tech, now being hired in AI startups because they are good bullshitters.

Countless directors and managers are now cosplaying as engineers. I've seen so many myself and that's just my tiny slice of this engineering world

It is pretty funny to see the shoe on the other foot, since it's usually software engineers with unearned arrogance about other fields.

Re: SQLite Critical CVEs or LLM Slop?

#260

Post by __s »

insanitybit wrote: post #96
inigyou wrote: post #51

Earlier quoted context omitted.

What do you mean? It isn't a DoS to assign every single bug fix a CVE! Every single bug is making someone vulnerable in some way.

That's nonsense. The goal is and always has been to undermine the CVE system because Greg does not believe in the system. The kernel is the only project that marks CVEs to fixes, it's the only project that considers every bug to be a "Security" bug, and it has a multi-decades long history of telling the security world to fuck off - all consistent with what I've said. No one else has the process that the kernel has, d…

Haha, I was reading your comment as praise for Greg until the end
Post reply on HN