How does it compare to Bifrost?
LiteLLM is basically Bifrost in the Python ecosystem.
Yeah, but bifrost seems tighter and claims to use way fewer resources
Litelm: LiteLLM Without the Bloat
21–30 of 73 posts
Re: Litelm: LiteLLM Without the Bloat
#22This is a 30 minute project with a frontier LLM. I don’t see why anyone would use anyone else’s router. Techniques are valuable today. Libraries are not.
Actually not. There are so many edge cases. Also these routers are only useful if they have a minimal layer of observability. Yes, LLMs can do a great job at writing semi-working MVP. Turning it into a usable project still requires a team. Yeah, maybe for your toy project you can use a LLM written tool. Also, I am not saying LiteLLM is good either.
There are always people who need an entire team to produce something like OP repo. Enterprise FizzBuzz is real after all.
Re: Litelm: LiteLLM Without the Bloat
#23I imagine many people code their own LLM client after getting fed up with the bad options out there. It’s very easy with ai coding tools.
I’m biased but I think mine is coded to a higher standard than litelm. https://github.com/s-banach/langchaint
Re: Litelm: LiteLLM Without the Bloat
#24Funny, everything you pruned away is the reason I’m deploying LiteLLM in our platform. Having a reliable way to track token spend per customer across different services is important to us, and LiteLLM handles this well
Re: Litelm: LiteLLM Without the Bloat
#25I strongly recommend the authors rewrite the readme by hand. It’s kind of a snif test for how much care someone put into this project.
[flagged]
1. Not a tangent, it’s related to the very first content visible on that link.
2. Not a dismissal, an advise from an expert
3. Not complaining, as stated, giving an advise about the optics of using clear LLM prose on the first paragraph
Re: Litelm: LiteLLM Without the Bloat
#26It would be great if there were a plugin/extension infrastructure. For example, to write the cache and costs however and whenever I see fit
Re: Litelm: LiteLLM Without the Bloat
#27Re: Litelm: LiteLLM Without the Bloat
#28This is a cool project, and the idea of using LLMs to selectively extract features from open source projects is an interesting concept. The only thing I take issue with is the phrase "LiteLLM Without the Bloat." A lot of the features that have been removed (like cost tracking, streaming, caching) are... kind of the core value proposition of LiteLLM for many of their users.
LiteLLM doesn't quite live up to its name. With all those features, there is nothing "lite" about it. It is essential for a project to live up to its name. Imagine Sqlite adding heavy features from Postgresql, e.g. row-level security.
We run it at my org and it's never been a noticeable resource hog. It's actually the best performer between it, our AI observability stack and the front end.
Re: Litelm: LiteLLM Without the Bloat
#29most software like this will be dematerialized, democratized, and demonetized - companies building in the infra band being increasingly disintermediated
Re: Litelm: LiteLLM Without the Bloat
#30I strongly recommend the authors rewrite the readme by hand. It’s kind of a snif test for how much care someone put into this project.
I have the same complaint about oh-my-pi's readme. The tone is obnoxious. It's somehow jaded and matter-of-fact at the same time. Like it was written by that one guy at work who never misses a chance to brag about how clever he is.