161–170 of 225 posts
Earlier quoted context omitted.
This message contains a lot of emotions and not too many coherent arguments. What did you actually want to say?
If you seek to audit code through the use of LLMs then you have inherently misunderstood the capabilities of the technology and will be left disappointed.
Earlier quoted context omitted.
I really don't care where you've worked, to seriously argue that LLMs aren't more capable of conversation than ELIZA, aren't capable of pair programming even, is gargantuan levels of cope.
I didn't make any claims about their utility. I said that they are not like people. They are machines through and through. Regular software programs. Programs that are, I suppose, a little bit too complex for the average human to understand, so now we have the Eliza effect applying to an entirely new generation. "I had not realized ... exposures to a relatively simple computer program could induce powerful delusional…
If this is the case and the latest models can be explained through their weights and settings, please link it. I would like to see explainable ai up and coming.
Earlier quoted context omitted.
fuck man, I'm either seriously stupid or y'all are taking crazy pills. LLMs are auto-complete on steroids; I've lived through enough iterations of Markov Chains giving semi-sensible output (that we give meaning to) and neural networks which present the illusion of intelligence to see directly what these LLMs are: a fuck load of compute designed to find "the next most common word" given the preceding 10,000 or more wo…
I wish people would move on from this mindset. "Agentic" workflows such as those implemented by Claude Code or Cursor can definitely reason about code, and I've used them to successfully debug small issues occurring in my codebase. We could argue about how they only "predict the next word", but there's also other stuff going on in the other layers of their NNs which do facilitate some sort of reasoning in the latent…
> I've used them to successfully debug small issues occurring in my codebase.
Great! The pattern recognition machine successfully identified pattern.
But, how do you know that it won't flag the repaired pattern because you've added a guard to prevent the behaviour (ie; invalid/out of bounds memory access guarded by a heavy assert on a sized object before even entering the function itself)?
What about patterns that aren't in the training data because humans have a hard time identifying the bad pattern reliably?
The point I'm making is that it's autocomplete; if your case is well covered it will show up: wether you have guards or not (so: noise) and that it will totally miss anything that humans haven't identified before.
It works: absolutely, but there's no reliability and that's sort of inherent in the design.
For security auditing specifically, an unreliable tool isn't just unhelpful: it's actively dangerous, because false confidence is actually worse than an understood ignorance
Planning serves two purposes - helping the model stay on track and helping the user gain confidence in what the model is about to do. Both sides of that are fuzzy, complex and non-obvious!
Here’s the original article which was much more informative and interesting: https://web.archive.org/web/20260314105751/https://backnotpr... Can’t believe HN has become so afraid of generic probably-unenforceable “plz don’t reverse engineer” EULAs. We deserve to know what these tools are doing. I’ve seen poor results from plan mode recently too and this explains a lot.
Seems like a straightforward solution would be to get people to opt-in by offering them credits, increased limits, early access to new features, etc. Universities have IRBs for good reasons.
A problem with this approach could be that you're now only testing the feature with the kind of people who would sign up for an A/B test. This group may not be representative of your whole user-base.
And unlike the university context, there’s a glut of data.
A basic technique: https://en.wikipedia.org/wiki/Inverse_probability_weighting
Or https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4384809
Hi, this was my test! The plan-mode prompt has been largely unchanged since the 3.x series models and now 4.x get models are able to be successful with far less direction. My hypothesis was that shortening the plan would decrease rate-limit hits while helping people still achieve similar outcomes. I ran a few variants, with the author (and few thousand others) getting the most aggressive, limiting the plan to 40 line…
The framing of A/B testing as a "silent experimentation on users" and invoking Meta is a little much. I don't believe A/B testing is an inherent evil, you need to get the test design right, and that would be better framing for the post imo. That being said, vastly reducing an LLMs effectiveness as part of an A/B test isn't acceptable which appears to be the case here.
Enshittification is coming for AI.