Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Any-LLM – Lightweight router to access any LLM Provider

github.com

Post reply on HN
Print view View original

71–72 of 72 posts

Re: Show HN: Any-LLM – Lightweight router to access any LLM Provider

#71

Post by gapeleon »

You guys need to fact check your AI-generated blog posts:

https://blog.mozilla.ai/introducing-any-llm-a-unified-api-to...

> One popular solution, LiteLLM, is highly valued for its wide support of different providers and modalities, making it a great choice for many developers. However, it re-implements provider interfaces rather than leveraging SDKs that are managed and released by the providers themselves. As a result, the approach can lead to compatibility issues and unexpected modifications in behavior, making it difficult to keep up with the changes happening among all the providers.

LiteLLM is rock-solid in practice. The underlying API providers announce breaking changes well in advance, and LiteLLM has never been caught out by this. LLMs will come up with hypothetical cons like this upon request.

> Lastly, proxy/gateway solutions like OpenRouter and Portkey require users to set up a hosted proxy server to act as an intermediary between their code and the LLM provider. Although this can effectively abstract away the complicated logic from the developer, it adds an extra layer of complexity and a dependency on external services, which might not be ideal for all use cases.

OpenRouter is a hosted service that provides the proxy/gateway infrastructure. Users don't "set up a hosted proxy server" themselves; they just make API calls to OpenRouter's endpoints. But older LLMs don't know what OpenRouter is and will assume it's a self-hosted proxy server.

> Another option, AISuite, was created by Andrew NG and offers a clean and modular design. However, it is not actively maintained (its last release was in December of 2024) and lacks consistent Python-typed interfaces.

Okay so you clicked the "releases" tab and saw December 2024. Next time check https://github.com/andrewyng/aisuite/commits/main/ Small, fast moving community projects like this, exllamav2, etc don't necessarily tag releases.

I've got nothing against using AI to write posts like this, but at least take the time to fact check before dumping on other people's work.

If not for the Mozilla branding, I'd have assumed this was a scam/malware - especially since it's name is so similar to Anything-LLM.

Re: Show HN: Any-LLM – Lightweight router to access any LLM Provider

#72

Post by RussianCow »

honorable_coder wrote: post #12
RussianCow wrote: post #10

Earlier quoted context omitted.

You can do all of that without a proxy. Just store the current state in your database or a Redis instance.

and managed from among the application servers that are greedily trying to store/retrieve this state? Not to mention you'll have to be in the business of defining, updating and managing the schema, ensuring that upgrades to the db don't break the application servers, etc, etc. The proxy server is the right design decision if you are truly trying to build something production worthy and you want it to scale.

> Not to mention you'll have to be in the business of defining, updating and managing the schema, ensuring that upgrades to the db don't break the application servers, etc, etc.

I have to do this already with practically all software I write, so the comexity is already baked in. Sure, if you don't already have a database or cache, maybe a proxy is simpler, but otherwise it's just extra infrastructure you need to manage.

> The proxy server is the right design decision if you are truly trying to build something production worthy and you want it to scale.

I've been doing stuff like the above (not for LLMs but similar use cases) for years "at scale" without issues. But in any case, you need to store state the moment you scale beyond a single proxy server anyway. Plus, most products never achieve a scale where this discussion matters.

Post reply on HN