251–260 of 378 posts
Earlier quoted context omitted.
What would this look like?
the model generates probabilities for the next token, then you set the probability of not allowed tokens to 0 before sampling (deterministically or probabilistically)
Earlier quoted context omitted.
So like every software? Why do you think there are so many security scanners and whatnot out there? There are millions of lines of code running on a typical box. Unless you're in embedded, you have no real idea what you're running.
...No, it's not at all "like every software". This seems like another instance of a problem I see so, so often in regard to LLMs: people observe the fact that LLMs are fundamentally nondeterministic, in ways that are not possible to truly predict or learn in any long-term way...and they equate that, mistakenly, to the fact that humans, other software, what have you sometimes make mistakes . In ways that are generally…
Are you really not seeing that GP is saying exactly this about LLMs?
What you want for this to be practical is verification and low enough error rate. Same as in any human-driven development process.
"This was a marathon session. I will congratulate myself endlessly on being so smart. We're in a good place to pick up again tomorrow."
"I'm not proceeding on feature X"
"Oh you're right, I'm being lazy about that."
Earlier quoted context omitted.
In this context, it means being able to deterministically predict properties of the output based on properties of the input. That is, you don’t treat each distinct input as a unicorn, but instead consider properties of the input, and you want to know useful properties of the output. With LLMs, you can only do that statistically at best, but not deterministically, in the sense of being able to know that whenever the i…
I mean can’t you have a grammar on both ends and just set out-of-language tokens to zero. I thought one of the APIs had a way to staple a JSON schema to the output, for ex. We’re making pretty strong statements here. It’s not like it’s impossible to make sure DROP TABLE doesn’t get output.
Everything to do with LLM prompts reminds me of people doing regexes to try and sanitise input against SQL injections a few decades ago, just papering over the flaw but without any guarantees. It's weird seeing people just adding a few more "REALLY REALLY REALLY REALLY DON'T DO THAT" to the prompt and hoping, to me it's just an unacceptable risk, and any system using these needs to treat the entire LLM as untrusted t…
The principal security problem of LLMs is that there is no architectural boundary between data and control paths. But this combination of data and control into a single, flexible data stream is also the defining strength of a LLM, so it can’t be taken away without also taking away the benefits.
Earlier quoted context omitted.
Language models are deterministic unless you add random input. Most inference tools add random input (the seed value) because it makes for a more interesting user experience, but that is not a fundamental property of LLMs. I suspect determinism is not the issue you mean to highlight.
Sort of. They are deterministic in the same way that flipping a coin is deterministic - predictable in principle, in practice too chaotic. Yes, you get the same predicted token every time for a given context. But why that token and not a different one? Too many factors to reliably abstract.
I was expecting some kind of explanation for this
I’ve hit this! In my otherwise wildly successful attempt to translate a Haskell codebase to Clojure [0], Claude at one point asks: [Claude:] Shall I commit this progress? [some details about what has been accomplished follow] Then several background commands finish (by timeout or completing); Claude Code sees this as my input, thinks I haven’t replied to its question, so it answers itself in my name: [Claude:] Yes, g…