Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: How do you add guard rails in LLM response without breaking streaming?

news.ycombinator.com

Post reply on HN
Print view View original

21–26 of 26 posts

Re: Ask HN: How do you add guard rails in LLM response without breaking streaming?

#21

Post by CharlieDigital »

lolinder wrote: post #15
CharlieDigital wrote: post #8

If it's the problem I think it is, the solution is to run two concurrent prompts. First prompt validates the input. Second prompt starts the actual content generation. Combine both streams with SSE on the front end and don't render the content stream result until the validation stream returns "OK". In the SSE, encode the chunks of each stream with a stream ID. You can also handle it on the server side by cancelling e…

This doesn't solve the critical problem, which is that you usually can't tell if something is okay until you have context that you don't yet have. This is why even SOTA models will backtrack when you hit the filter—they only realize you're treading into banned territory after a bunch of text has already been generated, including text that already breaks the rules. This is hard to fix because if you don't wait until y…

That's why I qualified it "general technique" and explicitly mentioned the option of server abort.

For most consumer use cases, it probably doesn't matter if a few tokens leak before the about, especially if they're not rendered.

Tune it to your needs :)

Re: Ask HN: How do you add guard rails in LLM response without breaking streaming?

#22

Post by tweezy »

I've tried a few things that seem to work. The first works pretty much perfectly, but adds quite a bit of latency to the final response. The second isn't perfect, but it's like 95% there

1 - the first option is to break this in to three prompts. The first prompt is either write a brief version, an outline of the full response, or even the full response. The second prompt is a validator, so you pass the output of the first to a prompt that says "does this follow the instructions. Return True | False." If True, send it to a third that says "Now rewrite this to answer the user's question." If False, send it back to the first with instructions to improve the response. This whole process can mean it takes 30 seconds or longer before the streaming of the final answer starts.

There are plenty of variations on the above process, so obviously feel free to experiment.

2 - The second option is to have instructions in your main prompt that says "Start each response with an internal dialogue wrapped in tags. Inside those tags first describe all of the rules you need to follow, then plan out exactly how you will respond to the user while following those rules."

Then on your frontend have the UI watch for those tags and hide everything between them from the user. This method isn't perfect, but it works extremely well in my experience. And if you're using a model like gpt-4o or claude 3.5 sonnet, it makes it really hard to make a mistake. This is the approach we're currently going with.

Re: Ask HN: How do you add guard rails in LLM response without breaking streaming?

#23

Post by vunderba »

CharlieDigital wrote: post #8

If it's the problem I think it is, the solution is to run two concurrent prompts. First prompt validates the input. Second prompt starts the actual content generation. Combine both streams with SSE on the front end and don't render the content stream result until the validation stream returns "OK". In the SSE, encode the chunks of each stream with a stream ID. You can also handle it on the server side by cancelling e…

The OP is talking about constraining the response not the input. Granted, in many cases, the input may give some kind of indicator that the large language model may be more prone to generating output that could violate the given constraints but this is not guaranteed by any measure.

As far as I know, there's no way of validating a streamed response until those tokens have already been streamed unfortunately. You could try buffering the stream in larger chunks before displaying them on screen in the hopes that you might be able to catch it earlier, but that's not going to be a great user experience either.

Post reply on HN