Earlier quoted context omitted.
To start with, it eliminates the IPC and Sidecar overhead of using something like Ollama. It might seem trivial when comparing the speed of a REST request to that of inferencing, but the ms add up at scale. And for what I'm doing higher up the stack, every nanosecond I save here, gives me headroom to do cooler things. It also allows you to make lots of smaller round trips to your AI generating layer. Even if you're j…
Thank you for the detailed and technical response. I’m not doubting it providers benefits at the technical level, and I’m keen to dive into it deeper, but I was hoping to understand what benefits this translates to at a user level. I’m imagining something like it can lead to faster switching between between agent calls, or even at a developer level, it allows you to run agents/llms in a more constrained environment,…
Your intuitions are right, faster switching between agent calls, and tighter packing of agents in the same amount of compute space.
There's a second thing besides just efficiency that I provide here however. At a developer-user level, it automagically provides JVM developers (not just java) a frictionless way to add AI into their stack with a single dependency.
It makes using AI in java as easy as it is in python.
I'm providing the [what I've found to be missing] layer for AI in java in the most efficient way I could.
I'm not sure if by user you meant developer (user of this code) or user as in end-user. Let me know if I still haven't answered your question fully yet!