Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Control LLM Spend and Access with any-LLM-gateway

blog.mozilla.ai

Post reply on HN
Print view View original

21–29 of 29 posts

Re: Control LLM Spend and Access with any-LLM-gateway

#21

Post by tomComb »

benatkin wrote: post #18
bitpush wrote: post #8

I'm conflicted on what Mozilla is doing here. On the one hand, it is nice that they are getting involved but com'on, dont you all have Firefox to work on? This is a classic case of an over enthusiastic engineer who says yes / raises hand to everything, but doesnt do any one thing properly. At some point, you have to sit down and tell them to focus on one thing and do it properly.

They're dumping competition on two other open source python libraries LiteLLM and simonw's llm. Unlike these two, Mozilla's any-llm doesn't have to make money. I'm sure simonw will be welcoming because he's a friendly kind of guy, but it might seem frustrating to LiteLLM which has a paid offering, for which they'd prefer organic competition rather than whatever magic 8 ball Mozilla uses.

I don’t think those are comparable. simonw's llm Has a python SDK, but it’s very much CLI first. Light LLM is very much about the SDK. You can wrap some agent SDK’s around it, like Gemini, but that’s for agents not work flows. I can’t really think of them as in the same category.

Re: Control LLM Spend and Access with any-LLM-gateway

#22

Post by sothatsit »

NeutralCrane wrote: post #19
sothatsit wrote: post #16

Earlier quoted context omitted.

We use LiteLLM and it is a bit of a dumpster fire of enterprise features and bugs. I can't even update the budget on keys in the UI (enterprise feature, although it may be a bug that it is marked as such). I can still update budgets through the API, but the API is a bit of a mess as well. Then we've ran into a lot of bugs like the UI DDOSing itself when the retry mechanism broke and it just started spamming API reque…

Are there other alternatives you have been looking at? I’m just getting started looking at these LLM gateways. I was under the impression that LiteLLM was pretty popular but you are not the only one here with negative things to say about it.

I am planning to try any-llm-gateway that this post is about. We don't need anything fancy, so it seems that this might cover our needs.

Re: Control LLM Spend and Access with any-LLM-gateway

#23

Post by smcleod »

NeutralCrane wrote: post #10
smcleod wrote: post #7

Interested to see how this stacks up against Bifrost (fast but many features paywalled) and LiteLLM Proxy (featureful but garbage code quality). Especially if it gets a web admin / reporting frontend and high availability.

We are just now looking into LLM Gateways and LiteLLM was one I was considering looking into. I’m curious to hear more about what makes the code quality garbage.

I've deployed LiteLLM proxy in a number of locations and we're looking to swap it out (probably to Bifrost), we've seen many bugs with it that never should have made it to a release. Most stem from poor code quality or what I'd classify as poor development practises. It's also slow, it doesn't scale well and adds a lot of latency.

Bugs include but are not limited to multiple ways budget limits aren't enforced, parameter handling issues, configuration / state mismatches etc...

What makes this worse is if you come to the devs with the problem, a solution and even a PR it's very difficult to get them to understand or action it - let alone see critical things like major budget blowouts as a priority.

Re: Control LLM Spend and Access with any-LLM-gateway

#25

Post by dbish »

bravura wrote: post #2

Thoughts on any-llm-gateway versus litellm-proxy? litellm is a great library, but one team using litellm-proxy reported having many issues with it to me. I haven't tried it yet.

What were the problems? I've been trying it out and haven't hit issues yet, but not using it at scale yet so I'm curious what to watch out for. I figure it's open source (MIT) so I can make changes as needed if there was anything particulary annoying.

Re: Control LLM Spend and Access with any-LLM-gateway

#26

Post by dbish »

smcleod wrote: post #23
NeutralCrane wrote: post #10

Earlier quoted context omitted.

We are just now looking into LLM Gateways and LiteLLM was one I was considering looking into. I’m curious to hear more about what makes the code quality garbage.

I've deployed LiteLLM proxy in a number of locations and we're looking to swap it out (probably to Bifrost), we've seen many bugs with it that never should have made it to a release. Most stem from poor code quality or what I'd classify as poor development practises. It's also slow, it doesn't scale well and adds a lot of latency. Bugs include but are not limited to multiple ways budget limits aren't enforced, parame…

What about forking it for your own use? Not worth it for the bugs you had fixes for?

Re: Control LLM Spend and Access with any-LLM-gateway

#27

Post by smcleod »

dbish wrote: post #26
smcleod wrote: post #23

Earlier quoted context omitted.

I've deployed LiteLLM proxy in a number of locations and we're looking to swap it out (probably to Bifrost), we've seen many bugs with it that never should have made it to a release. Most stem from poor code quality or what I'd classify as poor development practises. It's also slow, it doesn't scale well and adds a lot of latency. Bugs include but are not limited to multiple ways budget limits aren't enforced, parame…

What about forking it for your own use? Not worth it for the bugs you had fixes for?

Not worth the technical debt and architecture of the codebase. To be honest I'd sooner completely rewrite it in Golang/Rust or otherwise.

Re: Control LLM Spend and Access with any-LLM-gateway

#28

Post by Manouchehri »

sothatsit wrote: post #16
SOLAR_FIELDS wrote: post #12

Earlier quoted context omitted.

LiteLLM is one of the most popular solutions. You would self host the gateway

We use LiteLLM and it is a bit of a dumpster fire of enterprise features and bugs. I can't even update the budget on keys in the UI (enterprise feature, although it may be a bug that it is marked as such). I can still update budgets through the API, but the API is a bit of a mess as well. Then we've ran into a lot of bugs like the UI DDOSing itself when the retry mechanism broke and it just started spamming API reque…

LiteLLM was good in the early days. I ran into more features than bugs. Sadly in the past year or so, I run into more bugs than features.

Re: Control LLM Spend and Access with any-LLM-gateway

#29

Post by benatkin »

tomComb wrote: post #21
benatkin wrote: post #18

Earlier quoted context omitted.

They're dumping competition on two other open source python libraries LiteLLM and simonw's llm. Unlike these two, Mozilla's any-llm doesn't have to make money. I'm sure simonw will be welcoming because he's a friendly kind of guy, but it might seem frustrating to LiteLLM which has a paid offering, for which they'd prefer organic competition rather than whatever magic 8 ball Mozilla uses.

I don’t think those are comparable. simonw's llm Has a python SDK, but it’s very much CLI first. Light LLM is very much about the SDK. You can wrap some agent SDK’s around it, like Gemini, but that’s for agents not work flows. I can’t really think of them as in the same category.

llm is a lot about the Python API (or SDK) as well: https://llm.datasette.io/en/stable/python-api.html

It shows how to use it async or sync, and even handles using async in a sync context.

It's hard to write a good CLI without also writing most of a Python API, and llm went the rest of the way by documenting it. I think llm has the best docs of the Python API of the three.

Post reply on HN