Hi all,
I am trying to build a simple LLM bot and want to add guard rails so that the LLM responses are constrained.
I tried adjusting system prompt but the response does not always honour the instructions from prompt.
I can manually add validation on the response but then it breaks streaming and hence is visibly slower in response.
How are people handling this situation?
Ask HN: How do you add guard rails in LLM response without breaking streaming?
1–10 of 26 posts
Re: Ask HN: How do you add guard rails in LLM response without breaking streaming?
#2Not sure about the exact nature of your project, but for something similar I’ve worked on, I had success using a combination of custom stop words and streaming data with a bit of custom logic layered on top. By fine-tuning the stop words specific to the domain and applying filters in real-time as the data streams in, I was able to improve the response to users taste. Depending on your use case, adding logic to dynamically adjust stop words or contextually weight them might also help you.
Re: Ask HN: How do you add guard rails in LLM response without breaking streaming?
#3What's your stack? What type of response times are you looking for?
Re: Ask HN: How do you add guard rails in LLM response without breaking streaming?
#4> Hi all, I am trying to build a simple LLM bot and want to add guard rails so that the LLM responses are constrained.
Give examples of how the LLM should respond. Always give it a default response as well (e.g. "If the user response does not fall into any of these categories, say x").
> I can manually add validation on the response but then it breaks streaming and hence is visibly slower in response.
I've had this exact issue (streaming + JSON). Here's how I approached it: 1. Instruct the LLM to return the key "test" in its response. 2. Make the streaming call. 3. Build your JSON response as a string as you get chunks from the stream. 4. Once you detect "key" in that string, start sending all subsequent chunks wherever you need. 5. Once you get the end quotation, end the stream.
Re: Ask HN: How do you add guard rails in LLM response without breaking streaming?
#5have it format in yaml instead of json, incomplete yaml is still valid yaml
Re: Ask HN: How do you add guard rails in LLM response without breaking streaming?
#6Depending on what sort of constraints you need on your output, a custom token sampler, logit bias, or verifying it against a grammar could do the trick.
Re: Ask HN: How do you add guard rails in LLM response without breaking streaming?
#7Hi, I run the model serving team at Databricks. Usually you run regex filters, LLAMA Guard, etc on chunks at a time so you are still streaming but it's in batches of tokens rather than single tokens at a time. Hope that helps!
You could of course use us and get that out of the box if you have access to Databricks.
Re: Ask HN: How do you add guard rails in LLM response without breaking streaming?
#8If it's the problem I think it is, the solution is to run two concurrent prompts.
First prompt validates the input. Second prompt starts the actual content generation.
Combine both streams with SSE on the front end and don't render the content stream result until the validation stream returns "OK". In the SSE, encode the chunks of each stream with a stream ID. You can also handle it on the server side by cancelling execution once the first stream ends.
Generally, the experience is good because the validation prompt is shorter and faster to last (and only) token.
The SSE stream ends up like this:
data: ing|tomatoes
data: ing|basil
data: ste|3. Chop the
Re: Ask HN: How do you add guard rails in LLM response without breaking streaming?
#9Hi, I run the model serving team at Databricks. Usually you run regex filters, LLAMA Guard, etc on chunks at a time so you are still streaming but it's in batches of tokens rather than single tokens at a time. Hope that helps! You could of course use us and get that out of the box if you have access to Databricks.
But ultimately, it's an unsolved problem in the field. Every single LLM has been jailbroken.
Re: Ask HN: How do you add guard rails in LLM response without breaking streaming?
#10We are using keep-ai.com for a set of health care related AI project experiments.