There is liteLLM, OpenRouter, Arch (although that’s an edge/service proxy for agents) and now this. We all need a new problem to solve
And all of them despite 80% of model providers offering an OpenAI compatible endpoint
Show HN: Any-LLM – Lightweight router to access any LLM Provider
31–40 of 72 posts
Re: Show HN: Any-LLM – Lightweight router to access any LLM Provider
#32https://xkcd.com/927/ . LiteLLM rocks !
Re: Show HN: Any-LLM – Lightweight router to access any LLM Provider
#33> LiteLLM: While popular, it reimplements provider interfaces rather than leveraging official SDKs, which can lead to compatibility issues and unexpected behavior modifications with no vested interest in litellm, i'll challenge you on this one. what compatibility issues have come up? (i expect text to have the least, and probably voice etc have more but for text i've had no issues) you -want- to reimplement interface…
Yeah, official SDKs are sometimes a problem too. Together's included Apache Arrow, a ~60MB dependency, for a single feature (I patched to make it optional). If they ever lock dependency versions it could conflict with your project.
I'd rather a library that just used OpenAPI/REST, than one that takes a ton of dependencies.
Re: Show HN: Any-LLM – Lightweight router to access any LLM Provider
#34> LiteLLM: While popular, it reimplements provider interfaces rather than leveraging official SDKs, which can lead to compatibility issues and unexpected behavior modifications with no vested interest in litellm, i'll challenge you on this one. what compatibility issues have come up? (i expect text to have the least, and probably voice etc have more but for text i've had no issues) you -want- to reimplement interface…
LiteLLM is quite battle tested at this point as well. > it reimplements provider interfaces rather than leveraging official SDKs, which can lead to compatibility issues and unexpected behavior modifications Leveraging official SDKs also does not solve compatibility issues. any_llm would still need to maintain compatibility with those offical SDKs. I don't think one way clearly better than the other here.
That's true. We traded API compatibility work for SDK compatibility
work. Our bet is that providers are better at maintaining their own SDKs than we are at reimplementing their APIs. SDKs break less often and more predictably than APIs, plus we get provider-implemented features (retries, auth refresh, etc) "for free." Not zero maintenance, but definitely less. We use this in production at Mozilla.ai, so it'll stay actively maintained.
Re: Show HN: Any-LLM – Lightweight router to access any LLM Provider
#35> LiteLLM: While popular, it reimplements provider interfaces rather than leveraging official SDKs, which can lead to compatibility issues and unexpected behavior modifications with no vested interest in litellm, i'll challenge you on this one. what compatibility issues have come up? (i expect text to have the least, and probably voice etc have more but for text i've had no issues) you -want- to reimplement interface…
there is nothing lite in litellm ... i was experimenting (using as a lib) but ended using https://llm.datasette.io/en/stable/index.html btw. thanks @simonw for llm
Re: Show HN: Any-LLM – Lightweight router to access any LLM Provider
#36Anything like this, but in TypeScript?
Re: Show HN: Any-LLM – Lightweight router to access any LLM Provider
#37https://xkcd.com/927/ . LiteLLM rocks !
[deleted]
Re: Show HN: Any-LLM – Lightweight router to access any LLM Provider
#38There is liteLLM, OpenRouter, Arch (although that’s an edge/service proxy for agents) and now this. We all need a new problem to solve
portkey as well which is both js and open source https://www.latent.space/p/gateway
why provide link if there is not a single portkey keyword there?
Re: Show HN: Any-LLM – Lightweight router to access any LLM Provider
#39Anything like this, but in TypeScript?
Python only for now. Most providers have official TypeScript SDKs though, so the same approach (wrapping official SDKs) would work well in TS too.
Re: Show HN: Any-LLM – Lightweight router to access any LLM Provider
#40Interesting timing. Projects like Any-LLM or LiteLLM solve backend routing well but still involve server-side code. I’ve been tackling this from a different angle with Airbolt [1], which completely abstracts backend setup. Curious how others see the trade-offs between routing-focused tools and fully hosted backends like this. [1] https://github.com/Airbolt-AI/airbolt
(retracted after GP edited their comment)
I didn’t intend my original comment to be overly-promotional without relevance. I'm genuinely curious about the tradeoffs between different LLM API routing solutions, most acutely as a consumer.