11–20 of 41 posts
You say your rubric approach is “better than llm as a judge.” Can you please elaborate on what makes you say that?
How does Lucidic define the term "AI agent"?
Colloquially, AI agents are just while loops with LLM calls and tool calls. More specifically, what distinguishes an agent from LLM pipelines is that its next step is determined dynamically (based on the output of the previous one) so the execution path isn’t fixed. The boundary between complex LLM chaining and agents is pretty fuzzy, but we support both. Haha also our whole backend is in Django :)
But our stack is in Go and it has been tough to see a lot of observability tools focus on Python rather than an agnostic endpoint proxy like Helicone has.
I do feel frustrated with the current state of evaluations for long-lived sessions with many tool calls -- by default OpenAI's built-in eval system seems to rate chat completions that end with a tool call as "bad" because the tool call response is only in the next completion. But our stack is in Go and it has been tough to see a lot of observability tools focus on Python rather than an agnostic endpoint proxy like He…
Looks great! debugging agents is a huge pain for me, and this actually looks useful. Love the time travel and trajectory clustering ideas. Bookmarked to try it soon