Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Litelm: LiteLLM Without the Bloat

github.com

Post reply on HN
Print view View original

51–60 of 73 posts

Re: Litelm: LiteLLM Without the Bloat

#51

Post by blazarquasar »

mpyne wrote: post #28
OutOfHere wrote: post #6

Earlier quoted context omitted.

LiteLLM doesn't quite live up to its name. With all those features, there is nothing "lite" about it. It is essential for a project to live up to its name. Imagine Sqlite adding heavy features from Postgresql, e.g. row-level security.

We run it at my org and it's never been a noticeable resource hog. It's actually the best performer between it, our AI observability stack and the front end.

It may not be a huge resource hog, but it adds a ton of latency.

https://www.getmaxim.ai/bifrost/resources/benchmarks

Having ran both LiteLLM and Bifrost for months, I can largely confirm the numbers from those benchmarks for myself.

Re: Litelm: LiteLLM Without the Bloat

#53

Post by xg15 »

bityard wrote: post #30
khalic wrote: post #3

I strongly recommend the authors rewrite the readme by hand. It’s kind of a snif test for how much care someone put into this project.

I have the same complaint about oh-my-pi's readme. The tone is obnoxious. It's somehow jaded and matter-of-fact at the same time. Like it was written by that one guy at work who never misses a chance to brag about how clever he is.

One "AI-ism" I noticed is that LLMs often just put sentences behind one another without using any connecting adverbs - and just leave it to the reader's imagination how the sentences are related to each other.

E.g. in the readme: "LiteLLM routes LLM calls across providers and translates between message formats. That core is buried under 100k+ LOC of proxy servers, caching layers [etc...]".

Those two sentences have opposite sentiment on LiteLLM, so a human author would at least put a "but" between them. In contrast, the LLM just strings them together.

This reads "blunt" and "matter-of-fact" at first glance, but I wonder if it's really just an artifact of allocating less space for text generation and more for code in LLMs.

Re: Litelm: LiteLLM Without the Bloat

#55

Post by more-coffee »

raybb wrote: post #48
nylonstrung wrote: post #47

Earlier quoted context omitted.

IIRC it's 700mb with all deps included, to me that's clear bloat given you could accomplish it with a fraction of that

I guess they got that message so they're porting to rust! https://docs.litellm.ai/blog/litellm-rust-launch

That blog post smells like Opus.

Re: Litelm: LiteLLM Without the Bloat

#56

Post by otabdeveloper4 »

OutOfHere wrote: post #6
Centigonal wrote: post #4

This is a cool project, and the idea of using LLMs to selectively extract features from open source projects is an interesting concept. The only thing I take issue with is the phrase "LiteLLM Without the Bloat." A lot of the features that have been removed (like cost tracking, streaming, caching) are... kind of the core value proposition of LiteLLM for many of their users.

LiteLLM doesn't quite live up to its name. With all those features, there is nothing "lite" about it. It is essential for a project to live up to its name. Imagine Sqlite adding heavy features from Postgresql, e.g. row-level security.

LiteLLM is vibecoded trash.

Re: Litelm: LiteLLM Without the Bloat

#57

Post by otabdeveloper4 »

OutOfHere wrote: post #35
yujonglee wrote: post #33

Earlier quoted context omitted.

thanks for the feedback. genuinely curious what you think we could be doing better, especially around our dev practices. Would love to hear specifics.

Didn't yall have a release of a package that stole user keys or such?

Don't worry about it. Claude did a thorough security review of their code.

Re: Litelm: LiteLLM Without the Bloat

#58

Post by miki123211 »

yujonglee wrote: post #33
datadrivenangel wrote: post #16

Earlier quoted context omitted.

LiteLLM's problem isn't really features, it's how bloated all the features are, and specifically how AI maximalist and janky their dev practices are.

thanks for the feedback. genuinely curious what you think we could be doing better, especially around our dev practices. Would love to hear specifics.

Maybe this has improved since I last used it, but I was dismayed at not being able to do something like:

    litellm.register("foo", CustomAIProvider)
    litellm.do_whatever("foo/my-cool-model", "what is 2+2")

Re: Litelm: LiteLLM Without the Bloat

#59

Post by mark_l_watson »

I like the API, and since this is open source I will copy the design with attribution: I have been organically hacking little bits of LLM client code for most providers in Common Lisp for about 3 1/2 years and it is time to clean up all my old code. I probably need to do the same sort of refactoring for my search API wrappers.

Thanks for the cool project.

Re: Litelm: LiteLLM Without the Bloat

#60

Post by geek_at »

nylonstrung wrote: post #47
Centigonal wrote: post #4

This is a cool project, and the idea of using LLMs to selectively extract features from open source projects is an interesting concept. The only thing I take issue with is the phrase "LiteLLM Without the Bloat." A lot of the features that have been removed (like cost tracking, streaming, caching) are... kind of the core value proposition of LiteLLM for many of their users.

IIRC it's 700mb with all deps included, to me that's clear bloat given you could accomplish it with a fraction of that

And let's not forget that they regularly break things. For example they had merged a pull request that was supposed to fix issues related to the openrouter/free models but in the process broke openrouter for all models except the free ones.

The fix was deployed 2 weeks later (!!) to main (but you could downgrade of course).

Or that other time they broke model selection if you had selected "this key can used all models of their team" than the only model in the auto-selection for harnesses was an invalid "all-team-models" entry. Fixes this one in 1 week though.

all of them on the :latest docker tag btw

Post reply on HN