You start streaming the response immediately and kick off your guardrails checks. If the guard rail checks are triggered you cancel the streaming response.
Perfect is the enemy of good enough.
Ask HN: How do you add guard rails in LLM response without breaking streaming?
11–20 of 26 posts
Re: Ask HN: How do you add guard rails in LLM response without breaking streaming?
#12We have just the product for you! We’ve recently improved guardrail accuracy by 25% for a $5B client and would be happy to show you how we do it.
You're right - prompt eng. alone doesn't work. It's brittle and fails on most evals.
Ping me at shaunayrton@galini.ai
Re: Ask HN: How do you add guard rails in LLM response without breaking streaming?
#13Hi, I run the model serving team at Databricks. Usually you run regex filters, LLAMA Guard, etc on chunks at a time so you are still streaming but it's in batches of tokens rather than single tokens at a time. Hope that helps! You could of course use us and get that out of the box if you have access to Databricks.
But ultimately, it's an unsolved problem in the field. Every single LLM has been jailbroken.
Has o1 been jailbroken? My understanding is o1 is unique in that one model creates the initial output (chain of thought) then another model prepares the first response for viewing. Seems like that would be a fairly good way to prevent jailbreaks, but I haven't investigated myself.
Re: Ask HN: How do you add guard rails in LLM response without breaking streaming?
#14Earlier quoted context omitted.
But ultimately, it's an unsolved problem in the field. Every single LLM has been jailbroken.
Has o1 been jailbroken? My understanding is o1 is unique in that one model creates the initial output (chain of thought) then another model prepares the first response for viewing. Seems like that would be a fairly good way to prevent jailbreaks, but I haven't investigated myself.
Literally everything is trivial to jailbreak.
The core concept is to pass information into the model using a cipher. One that is not too hard that it can't figure it out, but not too easy as to be detected.
And yes, o1 was jailbroken shortly after release: https://x.com/elder_plinius/status/1834381507978280989
Re: Ask HN: How do you add guard rails in LLM response without breaking streaming?
#15If it's the problem I think it is, the solution is to run two concurrent prompts. First prompt validates the input. Second prompt starts the actual content generation. Combine both streams with SSE on the front end and don't render the content stream result until the validation stream returns "OK". In the SSE, encode the chunks of each stream with a stream ID. You can also handle it on the server side by cancelling e…
This doesn't solve the critical problem, which is that you usually can't tell if something is okay until you have context that you don't yet have. This is why even SOTA models will backtrack when you hit the filter—they only realize you're treading into banned territory after a bunch of text has already been generated, including text that already breaks the rules.
This is hard to fix because if you don't wait until you have enough context, you've given your censor a hair trigger.
> Combine both streams with SSE on the front end and don't render the content stream result until the validation stream returns "OK".
Just a note that this particular implementation has the additional problem of not actually applying your validation stream at the API level, which means your service can and will be abused worse than it would be if you combined the streams server-side. You should never rely on client-side validation for security or legal compliance.
Re: Ask HN: How do you add guard rails in LLM response without breaking streaming?
#16You start streaming the response immediately and kick off your guardrails checks. If the guard rail checks are triggered you cancel the streaming response. Perfect is the enemy of good enough.
About a year ago I was using an earlier version of Claude to help analyze the transcript from a podcast episode.
I'd fed in a raw transcript and I was asking it to do some basic editing, remove ums and ahs, that kind of thing.
It had streamed about 80% of the episode when it got to a bit where the podcast guest started talking about "bombing a data center"... and right in front of my eyes the entire transcript vanished. Claude effectively retracted the entire thing!
I tried again in a fresh window and hit Ctrl+A plus Ctrl+C while it was running to save as much as I could.
I don't think the latest version of Claude does that any more - if so, I've not seen it.
Re: Ask HN: How do you add guard rails in LLM response without breaking streaming?
#17fake it.
add some latency to the first token and then "stream" at the rate you received tokens even though the entire thing (or some sizable chunk) has been generated. that'll give you the buffer you need to seem fast while also staying safe.
Re: Ask HN: How do you add guard rails in LLM response without breaking streaming?
#18From what I can tell, ChatGPT appears to be doing "optimistic" streaming... It will start streaming the response to the user but may eventually hide the response if it trips some censorship filter. The user can theoretically capture the response from the network since the censorship is essentially done client-side but I guess they consider that good enough.
Re: Ask HN: How do you add guard rails in LLM response without breaking streaming?
#19If it's the problem I think it is, the solution is to run two concurrent prompts. First prompt validates the input. Second prompt starts the actual content generation. Combine both streams with SSE on the front end and don't render the content stream result until the validation stream returns "OK". In the SSE, encode the chunks of each stream with a stream ID. You can also handle it on the server side by cancelling e…
One of the things I love about gen ai is that all it's problems are solved with using more gen ai.
Re: Ask HN: How do you add guard rails in LLM response without breaking streaming?
#20Google has some safety feature in Vertex AI to block certain keywords, but that does break the streaming. If it finds something offending, it replaces with a static response. That is one that I have felt "works", but it is a bit wonky from UX side.