Looks like LangChain has LangSmith but it's in closed beta.
I saw a couple YC launches like Hegel AI.
I'm personally interested in deployments in small teams or teams with a lot of freedom to pick and choose their own tooling.
Ask HN: How are you managing LLM APIs in production?
1–9 of 9 posts
Re: Ask HN: How are you managing LLM APIs in production?
#2I'm currently writing up a deployment architecture for LLM's and the API question is answered here https://fine-tuna.com/docs/choosing-a-model/model/
Basically you can get a Docker container that will publish an Open AI API compatible end point. You can then choose the model that sits behind that API.
As deployment will be in Kuberenetes we will clusters with GPU resources to maxz out performance but we're not there yet.
Re: Ask HN: How are you managing LLM APIs in production?
#3Play around with langchain and then convert all of that into decent code. After a few prototypes, you'll realize langchains or other pipelining are just for non-coders. You can architect elegant solutions yourself.
Re: Ask HN: How are you managing LLM APIs in production?
#4I'm currently writing up a deployment architecture for LLM's and the API question is answered here https://fine-tuna.com/docs/choosing-a-model/model/ Basically you can get a Docker container that will publish an Open AI API compatible end point. You can then choose the model that sits behind that API. As deployment will be in Kuberenetes we will clusters with GPU resources to maxz out performance but we're not there…
Very cool!
Re: Ask HN: How are you managing LLM APIs in production?
#5Langsmith is broadly for tracing the chains - are you looking for prompt deployment solutions?
Re: Ask HN: How are you managing LLM APIs in production?
#6Langsmith is broadly for tracing the chains - are you looking for prompt deployment solutions?
Interesting - If I'm running a "single" chain, do you know if langsmith would be helpful?
Deployment solutions would be interesting - things are so new it would be hard to choose so my default would be just to wrap the API and keep it simple.
Re: Ask HN: How are you managing LLM APIs in production?
#7Play around with langchain and then convert all of that into decent code. After a few prototypes, you'll realize langchains or other pipelining are just for non-coders. You can architect elegant solutions yourself.
I'll give it a shot and keep that in mind
Re: Ask HN: How are you managing LLM APIs in production?
#8We built an AWS serverless app that handles:
- Configurable context and cases mapped to a RESTful API
- Multi-account and high throughput error handling
- DDB backed records of all requests and responses for evaluation, debugging & training
- One-click devops deploy
Has helped us deploy and maintain LLM apps into production quite easily. Let me know if you would like access to the repo.
Re: Ask HN: How are you managing LLM APIs in production?
#9Langsmith is broadly for tracing the chains - are you looking for prompt deployment solutions?
Interesting - If I'm running a "single" chain, do you know if langsmith would be helpful? Deployment solutions would be interesting - things are so new it would be hard to choose so my default would be just to wrap the API and keep it simple.
Langsmith would be useful in that case to see the traces for your chain. (Of course given that you are using Langchain for chaining the prompts)
Makes sense on the deployment thing. I like what Vellum is doing. Helicone and Portkey also let you do deployments of prompt templates through APIs.