Anything like this, but in TypeScript?
ai-sdk by vercel?
Show HN: Any-LLM – Lightweight router to access any LLM Provider
41–50 of 72 posts
Re: Show HN: Any-LLM – Lightweight router to access any LLM Provider
#42> LiteLLM: While popular, it reimplements provider interfaces rather than leveraging official SDKs, which can lead to compatibility issues and unexpected behavior modifications with no vested interest in litellm, i'll challenge you on this one. what compatibility issues have come up? (i expect text to have the least, and probably voice etc have more but for text i've had no issues) you -want- to reimplement interface…
I use litellm as my personal AI gateway, and from user point of view there is no difference if proxy uses official SDK or not, this might be benefit for proxy developers.
but I can give you one example: litellm recently had issue with handling deepseek reasoning. they broke implementation and while reasoning was missing from sync and streaming responses.
Re: Show HN: Any-LLM – Lightweight router to access any LLM Provider
#43https://xkcd.com/927/ . LiteLLM rocks !
I didn't even need to click the link to know what this comic was. LiteLLM is great, we just needed something slightly different for our use case.
Re: Show HN: Any-LLM – Lightweight router to access any LLM Provider
#44This looks awesome. Why Python? Probably because most of the SDKs are python, but something that could be ported across languages without requiring an interpreter would have been really amazing.
we essentially built the gateway as a service rather than an SDK: https://github.com/portkey-AI/gateway
Re: Show HN: Any-LLM – Lightweight router to access any LLM Provider
#45Anything like this, but in TypeScript?
there's portkey that we've been working on: https://github.com/portkey-AI/gateway
Re: Show HN: Any-LLM – Lightweight router to access any LLM Provider
#46> LiteLLM: While popular, it reimplements provider interfaces rather than leveraging official SDKs, which can lead to compatibility issues and unexpected behavior modifications with no vested interest in litellm, i'll challenge you on this one. what compatibility issues have come up? (i expect text to have the least, and probably voice etc have more but for text i've had no issues) you -want- to reimplement interface…
Both approaches work well for standard text completion. Issues tend to be around edge cases like streaming behavior, timeout handling, or new features rolling out.
You're absolutely right that any router reimplements interfaces for normalization. The difference is what layer we reimplement at. We use SDKs where available for HTTP/auth/retries and reimplement normalization.
Bottom line is we both reimplement interfaces, just at different layers. Our bet on SDKs is mostly about maintenance preferences, not some fundamental flaw in LiteLLM's approach.
Re: Show HN: Any-LLM – Lightweight router to access any LLM Provider
#47Earlier quoted context omitted.
I definitely appreciate all the work that has gone in to LiteLLM but it doesn't take much browsing through the 7000+ line `utils.py` to see where using it could become problematic ( https://github.com/BerriAI/litellm/blob/main/litellm/utils.p... )
can you double click a little bit? many files in professional repos are 1000s of lines. LoC in it self is not a code smell.
LiteLLM is the worst code I have ever read in my life. Quite an accomplishment, lol.
Re: Show HN: Any-LLM – Lightweight router to access any LLM Provider
#48Earlier quoted context omitted.
How do I put this behind a proxy? You mean run the module as a containerized service? But provider switching is built in some of these - and the folks behind envoy built: https://github.com/katanemo/archgw - developers can use an OpenAI client to call any model, offers preference-aligned intelligent routing to LLMs based on usage scenarios that developers can define, and acts as an edge proxy too.
To clarify: any-llm is just a Python library you import, not a service to run. When I said "put it behind a proxy," I meant your app (which imports any-llm) can run behind a normal proxy setup. You're right that archgw handles routing at the infrastructure level, which is perfect for centralized control. any-llm simply gives you the option to handle routing in your application code when that makes sense (For example,…
But you can also use tokens to implement routing decisions in a proxy. You can make RBAC natively available to all agents outside code. The incremental feature work in code vs an out of process server is the trade off. One gets you going super fast the other offers a design choice that (I think) scales a lot better
Re: Show HN: Any-LLM – Lightweight router to access any LLM Provider
#49How does this differ from this project? https://github.com/simonw/llm
Re: Show HN: Any-LLM – Lightweight router to access any LLM Provider
#50Crazy timing!
I shipped a similar abstraction for llms a bit over a week ago:
https://github.com/omarkamali/borgllm
pip install borgllm
I focused on making it Langchain compatible so you could drop it in as a replacement. And it offers virtual providers for automatic fallback when you reach rate limits and so on.