Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building your own CLI coding agent with Pydantic-AI

martinfowler.com

Post reply on HN
Print view View original

21–30 of 43 posts

Re: Building your own CLI coding agent with Pydantic-AI

#21

Post by DouweM »

photonthug wrote: post #20
DouweM wrote: post #18

Earlier quoted context omitted.

> All I know is that with the same LLM models, `openai.client.chat.completions` + a custom prompt to pass in the pydantic JSON schema + post-processing to instantiate SomePydanticModel(*json) creates objects successfully whereas vanilla pydantic-ai rarely does, regardless of the number of retries. That's very odd, would you mind sharing the Pydantic model / schema so I can have a look? (I'm a maintainer) What you're…

Thanks for the reply. Native output is indeed what I'm shooting for. I can't share the model directly right now, but putting together a min-repro and moving towards and actual bug report is something on todo list. One thing I can say though.. my models differ from the docs examples mostly in that they are not "flat" with simple top-level data structures. They have lots of nested models-as-fields.

Thanks, a reproducible example would be very useful. Note that earlier this month I made Pydantic AI try a lot harder to use strict JSON mode (in response to feedback from Python creator Guido of all people: https://github.com/pydantic/pydantic-ai/issues/2405), so if you haven't tried it in a little while, the problem you were seeing may very well have been fixed already!

Re: Building your own CLI coding agent with Pydantic-AI

#22

Post by photonthug »

DouweM wrote: post #21
photonthug wrote: post #20

Earlier quoted context omitted.

Thanks for the reply. Native output is indeed what I'm shooting for. I can't share the model directly right now, but putting together a min-repro and moving towards and actual bug report is something on todo list. One thing I can say though.. my models differ from the docs examples mostly in that they are not "flat" with simple top-level data structures. They have lots of nested models-as-fields.

Thanks, a reproducible example would be very useful. Note that earlier this month I made Pydantic AI try a lot harder to use strict JSON mode (in response to feedback from Python creator Guido of all people: https://github.com/pydantic/pydantic-ai/issues/2405 ), so if you haven't tried it in a little while, the problem you were seeing may very well have been fixed already!

> https://github.com/pydantic/pydantic-ai/issues/2405

Thanks, this is a very interesting thread on multiple levels. It does seem related to my problem and I also learned about field docstrings :) I'll try moving my dependency closer to the bleeding edge

Re: Building your own CLI coding agent with Pydantic-AI

#24

Post by __MatrixMan__ »

gck1 wrote: post #23

Curiously, I explicitly tell all my LLM agents to never touch Pydantic models or environment stuff - it’s even in big, uppercase, bold text in my custom instructions for Roo-Code. LLMs seem to trip a lot over Pydantic’s magic.

pydantic-AI is a bit different than pydantic. The LLM isn't prompted to generate the pydantic model, instead it's encouraged to take input in the form of one model and produce output in the form of another.

Re: Building your own CLI coding agent with Pydantic-AI

#25

Post by __MatrixMan__ »

DouweM wrote: post #15
iLoveOncall wrote: post #6

I really wish Pydantic invested in... Pydantic, instead of some AI API wrapper garbage. I've been using it a lot lately and anything beyond basic usage is an absolute chore.

I'm curious what issues you've run into, do you happen to have GitHub links so I can have a look? (I'm a maintainer.) Pydantic still sees multiple commits per week, which is less than it was at one point, but I'd say that's a sign of its maturity and stability more than a lack of attention.

My experience is that pretty frequently the LLM just refuses to actually supply json conforming to the model and summarizes the input instead. Even with several retries configured I still couldn't rely on it. I only spent an afternoon with it though so it's possible I'm just doing it wrong (either in how I'm prompting or in how I'm configuring pydantic-ai).

Re: Building your own CLI coding agent with Pydantic-AI

#26

Post by DouweM »

__MatrixMan__ wrote: post #25
DouweM wrote: post #15

Earlier quoted context omitted.

I'm curious what issues you've run into, do you happen to have GitHub links so I can have a look? (I'm a maintainer.) Pydantic still sees multiple commits per week, which is less than it was at one point, but I'd say that's a sign of its maturity and stability more than a lack of attention.

My experience is that pretty frequently the LLM just refuses to actually supply json conforming to the model and summarizes the input instead. Even with several retries configured I still couldn't rely on it. I only spent an afternoon with it though so it's possible I'm just doing it wrong (either in how I'm prompting or in how I'm configuring pydantic-ai).

How recently was that? I made a few improvements earlier this month: https://news.ycombinator.com/item?id=45058214

If the issue is still showing on the latest version, seeing the Pydantic model/schema would be very helpful.

Re: Building your own CLI coding agent with Pydantic-AI

#27

Post by venkyk »

bluecoconut wrote: post #4

After maintaining my own agents library for a while, I’ve switched over to pydantic ai recently. I have some minor nits, but overall it's been working great for me. I’ve especially liked combining it with langfuse. Towards coding agents, I wonder if there are any good / efficient ways to measure how much different implementations work on coding? SWE-bench seems good, but expensive to run. Effectively I’m curious for…

https://www.tbench.ai/ the article also refers to this benchmark

Re: Building your own CLI coding agent with Pydantic-AI

#28

Post by yahoozoo »

Am I correct in thinking that it would cost more if you used your own agent with Sonnet 4 than going through Claude Code since you would have to go through the Anthropic API? What models do folks with custom agents usually use? And what kind of prompts seem to provide the same responses that Claude Code would give you?

Re: Building your own CLI coding agent with Pydantic-AI

#29

Post by qianli_cs »

I've been building an integration [1] with Pydantic AI and the experience has been great. Questions usually get answered within a few hours, and the team is super responsive and supportive for external contributors. The public API is easy to extend for new functionality (in my case, durable agents).

Its agent model feels similar to OpenAI's: flexible and dynamic without needing to predefine a DAG. Execution is automatically traced and can be exported to Logfire, which makes observability pretty smooth too. Looking forward to their upcoming V1 release.

Shameless plug: I've been working on a DBOS [2] integration into Pydantic-AI as a lightweight durable agent solution.

[1] https://github.com/pydantic/pydantic-ai/pull/2638

[2] https://github.com/dbos-inc/dbos-transact-py

Re: Building your own CLI coding agent with Pydantic-AI

#30

Post by __MatrixMan__ »

DouweM wrote: post #26
__MatrixMan__ wrote: post #25

Earlier quoted context omitted.

My experience is that pretty frequently the LLM just refuses to actually supply json conforming to the model and summarizes the input instead. Even with several retries configured I still couldn't rely on it. I only spent an afternoon with it though so it's possible I'm just doing it wrong (either in how I'm prompting or in how I'm configuring pydantic-ai).

How recently was that? I made a few improvements earlier this month: https://news.ycombinator.com/item?id=45058214 If the issue is still showing on the latest version, seeing the Pydantic model/schema would be very helpful.

It was about a month ago. I'll take another swing at it and make an issue if I can't overcome it.

Thanks for being a proactive kind of maintainer. The world is better because of people like you.

Post reply on HN