I'm excited to see this. Have been using LiteLLM but it's honestly a huge mess once you peek under the hood, and it's being developed very iteratively and not very carefully. For example. for several months recently (haven't checked in ~a month though), their Ollama structured outputs were completely botched and just straight up broken. Docs are a hot mess, etc.
Show HN: Any-LLM – Lightweight router to access any LLM Provider
21–30 of 72 posts
Re: Show HN: Any-LLM – Lightweight router to access any LLM Provider
#22Interesting timing. Projects like Any-LLM or LiteLLM solve backend routing well but still involve server-side code. I’ve been tackling this from a different angle with Airbolt [1], which completely abstracts backend setup. Curious how others see the trade-offs between routing-focused tools and fully hosted backends like this. [1] https://github.com/Airbolt-AI/airbolt
(retracted after GP edited their comment)
Re: Show HN: Any-LLM – Lightweight router to access any LLM Provider
#23> LiteLLM: While popular, it reimplements provider interfaces rather than leveraging official SDKs, which can lead to compatibility issues and unexpected behavior modifications with no vested interest in litellm, i'll challenge you on this one. what compatibility issues have come up? (i expect text to have the least, and probably voice etc have more but for text i've had no issues) you -want- to reimplement interface…
LiteLLM is quite battle tested at this point as well. > it reimplements provider interfaces rather than leveraging official SDKs, which can lead to compatibility issues and unexpected behavior modifications Leveraging official SDKs also does not solve compatibility issues. any_llm would still need to maintain compatibility with those offical SDKs. I don't think one way clearly better than the other here.
Being battle tested is the only good thing I can say about LiteLLM.
Re: Show HN: Any-LLM – Lightweight router to access any LLM Provider
#24Earlier quoted context omitted.
Good points! any-llm handles the LLM routing, but you can still put it behind your own proxy for centralized control. We just don't force that architectural decision on you. Think of it as composable: use any-llm for provider switching, add nginx/envoy/whatever for rate limiting if you need it.
How do I put this behind a proxy? You mean run the module as a containerized service? But provider switching is built in some of these - and the folks behind envoy built: https://github.com/katanemo/archgw - developers can use an OpenAI client to call any model, offers preference-aligned intelligent routing to LLMs based on usage scenarios that developers can define, and acts as an edge proxy too.
To clarify: any-llm is just a Python library you import, not a service to run. When I said "put it behind a proxy," I meant your app (which imports any-llm) can run behind a normal proxy setup.
You're right that archgw handles routing at the infrastructure level, which is perfect for centralized control. any-llm simply gives you the option to handle routing in your application code when that makes sense (For example, premium users get Opus-4). We leave the architectural choice to you, whether that's adding a proxy, keeping routing in your app, or using both, or just using any-llm directly.
Re: Show HN: Any-LLM – Lightweight router to access any LLM Provider
#25I have been also working on very similar open source project for python llm abstraction layer. I needed one for my research job. I inspired from that and created one for more generic usage.
Github: https://github.com/proxai/proxai
Website: https://proxai.co/
Re: Show HN: Any-LLM – Lightweight router to access any LLM Provider
#26Earlier quoted context omitted.
LiteLLM is quite battle tested at this point as well. > it reimplements provider interfaces rather than leveraging official SDKs, which can lead to compatibility issues and unexpected behavior modifications Leveraging official SDKs also does not solve compatibility issues. any_llm would still need to maintain compatibility with those offical SDKs. I don't think one way clearly better than the other here.
Being battle tested is the only good thing I can say about LiteLLM.
You can add in it's still 10x better than LangChain
Re: Show HN: Any-LLM – Lightweight router to access any LLM Provider
#27This is awesome, will give it a try tonight.
I’ve been looking for something a bit different though related to Ollama. I’d like a load balancing reverse proxy that supports queuing requests to multiple Ollama servers and sending requests only when a Ollama server is up and idle (not processing). Anything exist?
Re: Show HN: Any-LLM – Lightweight router to access any LLM Provider
#28Interesting timing. Projects like Any-LLM or LiteLLM solve backend routing well but still involve server-side code. I’ve been tackling this from a different angle with Airbolt [1], which completely abstracts backend setup. Curious how others see the trade-offs between routing-focused tools and fully hosted backends like this. [1] https://github.com/Airbolt-AI/airbolt
(retracted after GP edited their comment)
don't you post links to your own stuff all the time? i don't think their comment was out of line.
Re: Show HN: Any-LLM – Lightweight router to access any LLM Provider
#29Earlier quoted context omitted.
LiteLLM is kind of a mess TBH, I guess it's ok if you just want a docker container to proxy to for personal projects, but actually using it in production isn't great.
I definitely appreciate all the work that has gone in to LiteLLM but it doesn't take much browsing through the 7000+ line `utils.py` to see where using it could become problematic ( https://github.com/BerriAI/litellm/blob/main/litellm/utils.p... )
can you double click a little bit? many files in professional repos are 1000s of lines. LoC in it self is not a code smell.
Re: Show HN: Any-LLM – Lightweight router to access any LLM Provider
#30This looks awesome. Why Python? Probably because most of the SDKs are python, but something that could be ported across languages without requiring an interpreter would have been really amazing.
for js/ts you have vercel aisdk [0], for c++ you have [1], for flutter/reactnative/kotlin there is [2]
[0] https://github.com/vercel/ai