What local models will it work with ? Also what will be the pricing for local llms?
Show HN: Butter – A Behavior Cache for LLMs
11–20 of 25 posts
Re: Show HN: Butter – A Behavior Cache for LLMs
#12interesting. is the answer not context specific most of the time? even if I ask LLM the same question again and again the answer depends on the context.
what are some use cases where you need deterministic caching?
Re: Show HN: Butter – A Behavior Cache for LLMs
#13We often will repeat calls to try again. Or sometimes we make the same call multiple times to get multiple answers and then score or merge them.
Is this used only in cases where you assume the answer from your first call is correct?
Re: Show HN: Butter – A Behavior Cache for LLMs
#14I also did computer agents with a vc backed startup, ran into the same issues, and we built a fairly similar thing at one point.
It’s useful but it has limitations, it seems to only work well in environments that are perfectly predictable otherwise it gets in the way of the agent.
I think I prefer RL over these approaches but it requires a bit more data.
Re: Show HN: Butter – A Behavior Cache for LLMs
#15We spoke to a number of browser agent companies who said deterministic RPA with an AI fallback was their "secret" :)
Re: Show HN: Butter – A Behavior Cache for LLMs
#16We spoke to a number of browser agent companies who said deterministic RPA with an AI fallback was their "secret" :)
Very, very common approach!
Wrote more on that here: https://blog.butter.dev/the-messy-world-of-deterministic-age...
Re: Show HN: Butter – A Behavior Cache for LLMs
#17We often will repeat calls to try again. Or sometimes we make the same call multiple times to get multiple answers and then score or merge them. Is this used only in cases where you assume the answer from your first call is correct?
I’d love your opinion here!
Right now, we assume first call is correct, and will eagerly take the first match we find while traversing the tree.
One of the worst things that could currently happen is we cache a bad run, and now instead of occasional failures you’re given 100% failures.
A few approaches we’ve considered - maintain a staging tree, and only promote to live if multiple sibling nodes (messages) look similar enough. Decision to promote could be via tempting, regex, fuzzy, semantic, or LLM-judged - add some feedback APIs for a client to score end-to-end runs so that path could develop some reputation
Re: Show HN: Butter – A Behavior Cache for LLMs
#18Earlier quoted context omitted.
I couldn’t see how it wouldn’t be, as it’s a free market opt-in decision to use Butter
it wouldn't be the first API service to disallow someone from selling a cache layer for their API. After all, this should likely result in OpenAI (or whatever provider) making less money
Ah yes that makes sense, have heard of those cases too but hadn’t put much thought into it. Thanks for pointing it out!
Re: Show HN: Butter – A Behavior Cache for LLMs
#19What local models will it work with ? Also what will be the pricing for local llms?
Good question, I imagine you’d need to set up an ngrok endpoint to tunnel to local LLMs.
In those cases perhaps an open source (maybe even local) version would make more sense. For our hosted version we’d need to charge something, given storage requirements to run such a service, but especially for local models that feels wrong. I’ve been considering open source for this reason.
Re: Show HN: Butter – A Behavior Cache for LLMs
#20Earlier quoted context omitted.
it wouldn't be the first API service to disallow someone from selling a cache layer for their API. After all, this should likely result in OpenAI (or whatever provider) making less money
Ah yes that makes sense, have heard of those cases too but hadn’t put much thought into it. Thanks for pointing it out!
I've seen the OpenRouter guys here on HN before, so you can probably ask them what to look out for.