AI proxy middlewares are a hack
blog.lastmileai.dev
AI proxy middlewares are a hack
1–4 of 4 posts
Re: AI proxy middlewares are a hack
#2I'm glad to see more focus on middlewares. Getting the architecture right is crucial to building effective models, but this has largely been overlooked as shown by the inefficiencies in many models today
Re: AI proxy middlewares are a hack
#3I'm glad to see more focus on middlewares. Getting the architecture right is crucial to building effective models, but this has largely been overlooked as shown by the inefficiencies in many models today
Agreed. The author makes the point that middleware is trying to solve valid problems. I think the issue is that there are better ways to address it by “avoiding the middleware all together”. Especially since it introduces other 3rd parties into the mix
Re: AI proxy middlewares are a hack
#4I'm glad to see more focus on middlewares. Getting the architecture right is crucial to building effective models, but this has largely been overlooked as shown by the inefficiencies in many models today
Agreed. The author makes the point that middleware is trying to solve valid problems. I think the issue is that there are better ways to address it by “avoiding the middleware all together”. Especially since it introduces other 3rd parties into the mix
Yea, half of this seems like problems we've already solved for APIs generally, the other half is LLM specific, like prompt management, log/mon, response quality in prod, real time feedback
It seems like tools from our current ops ought to work just fine for the non-LLM uniqueness. At the same time, Datadog is pretty popular for a managed experience, and the LLM services as proxies kind of fit that model
This project looks great and aligns with my thinking