Excited to try this
Launch HN: Lucidic (YC W25) – Debug, test, and evaluate AI agents in production
31–40 of 41 posts
Re: Launch HN: Lucidic (YC W25) – Debug, test, and evaluate AI agents in production
#32Earlier quoted context omitted.
Gotcha, you're using the "LLM calling tools in a loop" definition. I think that's a decent one, but I worry that many people out there are carrying around completely different ideas as to what the term means.
Do you have a writeup on the different interpretations of "AI Agent"?
I need to put one together. The big ones are:
- "LLM running tools in a loop" - often used by Anthropic, generally the most popular among software engineers who build things
- "An AI system that performs tasks on your behalf" - used by OpenAI, I dislike how vague this one is
- "an entity that perceives its environment through sensors and acts upon that environment through actuators to achieve specific goals" - the classic academic one, Russell and Norvig. I sometimes call this the "thermostat definition".
- "kinda like a travel agent I guess?" - quite common among less technical people I've talked to
I gathered over a hundred on Twitter last year, summarized by Gemini here: https://gist.github.com/simonw/beaa5f90133b30724c5cc1c4008d0...
I also have a tag about this on my blog: https://simonwillison.net/tags/agent-definitions/
Re: Launch HN: Lucidic (YC W25) – Debug, test, and evaluate AI agents in production
#33Love the UX. From the value POV, I am yet to see/experience how it differs from competitors.
P.S. I currently use Braintrust and Opik
Re: Launch HN: Lucidic (YC W25) – Debug, test, and evaluate AI agents in production
#34Earlier quoted context omitted.
Do you have a writeup on the different interpretations of "AI Agent"?
I need to put one together. The big ones are: - "LLM running tools in a loop" - often used by Anthropic, generally the most popular among software engineers who build things - "An AI system that performs tasks on your behalf" - used by OpenAI, I dislike how vague this one is - "an entity that perceives its environment through sensors and acts upon that environment through actuators to achieve specific goals" - the cl…
The way I draw the line is to focus on the "agency" aspect.
In workflows/pipelines the "agency" belongs to the coder/creator of the workflow. It usually resembles something like a "list of steps" or "ittt". Examples include traditional "research" flows like 1. create search terms for query; 2. search; 3. fetch_urls; 4. summarise; 5. answer
In agents the "agency" belongs, at one point or another, to the LLM. It gets to decide what to do at some steps, based on context, tools available, and actions taken. It usually resembles a loop, without predefined steps (or with vague steps like "if this looks like a bad answer, retry" - where bad answer can be another LLM invocation w/ a specific prompt). Example: Fix this ticket in this codebase -> ok, first I need to read_files -> read_files tool call ... and so on.
Re: Launch HN: Lucidic (YC W25) – Debug, test, and evaluate AI agents in production
#35Earlier quoted context omitted.
I need to put one together. The big ones are: - "LLM running tools in a loop" - often used by Anthropic, generally the most popular among software engineers who build things - "An AI system that performs tasks on your behalf" - used by OpenAI, I dislike how vague this one is - "an entity that perceives its environment through sensors and acts upon that environment through actuators to achieve specific goals" - the cl…
The way I draw the line is to focus on the "agency" aspect. In workflows/pipelines the "agency" belongs to the coder/creator of the workflow. It usually resembles something like a "list of steps" or "ittt". Examples include traditional "research" flows like 1. create search terms for query; 2. search; 3. fetch_urls; 4. summarise; 5. answer In agents the "agency" belongs, at one point or another, to the LLM. It gets t…
In the research workflow example, what if the first set of search queries don’t return good results. If the LLM tool loop decides to refine the queries, would this be “agency”?
Re: Launch HN: Lucidic (YC W25) – Debug, test, and evaluate AI agents in production
#36Earlier quoted context omitted.
The way I draw the line is to focus on the "agency" aspect. In workflows/pipelines the "agency" belongs to the coder/creator of the workflow. It usually resembles something like a "list of steps" or "ittt". Examples include traditional "research" flows like 1. create search terms for query; 2. search; 3. fetch_urls; 4. summarise; 5. answer In agents the "agency" belongs, at one point or another, to the LLM. It gets t…
In the research workflow example, what if the first set of search queries don’t return good results. If the LLM tool loop decides to refine the queries, would this be “agency”?
I'd say so, yeah. If the LLM "decides" what steps to take, that's an agent. If the flow is "hardcoded" then it's a workflow/pipeline. It often gets confused because early frameworks called these workflows/pipelines "agents".
Re: Launch HN: Lucidic (YC W25) – Debug, test, and evaluate AI agents in production
#37Congrats on the launch! On a tangential note, is this work open source or do you guys have some technical report that you could share? I am specially interested in your results on the clustering methods for surfacing behavioural patterns. Thanks!
We're new to the open source scene so we don't have anything published yet but plan to in the future. A basic overview of the way we do clustering is we condense stateful information -> create a state embedding -> create tags -> cluster based on distance of tags + embeddings.
Feel free to reach out if you want some guidance. At a minimum your SDK should be open source since it potentially touches sensitive data and you’ll want to build trust. Also, it probably technically already is unless you’ve only released Python binary wheels.
Re: Launch HN: Lucidic (YC W25) – Debug, test, and evaluate AI agents in production
#38Earlier quoted context omitted.
In the research workflow example, what if the first set of search queries don’t return good results. If the LLM tool loop decides to refine the queries, would this be “agency”?
I'd say so, yeah. If the LLM "decides" what steps to take, that's an agent. If the flow is "hardcoded" then it's a workflow/pipeline. It often gets confused because early frameworks called these workflows/pipelines "agents".
I see, that's a good way to think about it
Re: Launch HN: Lucidic (YC W25) – Debug, test, and evaluate AI agents in production
#39I'm looking into a tool like this for my startup. Why should I use this over Langfuse or Helicone?
Langfuse and Helicone work well for traditional LLM operations, but AI agents are different. We discovered that AI agents require fundamentally different tooling, here are some examples. First, while LLMs simply respond to prompts, agents often get stuck in behavioral loops where they repeat the same actions; to address this, we built a graph visualization that automatically detects when an agent reaches the same sta…
This makes sense. We'll look into this some more, will be making a decision next couple days :)
Good luck!
Re: Launch HN: Lucidic (YC W25) – Debug, test, and evaluate AI agents in production
#40What kinds of rubrics can one specify? Is there a tutorial or a page with some examples of this kind of rubrics defined in Lucidic?