Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building reliable agentic AI systems

martinfowler.com

Post reply on HN
Print view View original

11–20 of 74 posts

Re: Building reliable agentic AI systems

#11

Post by bob1029 »

The most important part is the database that the agent can see and how clean the data is. I pitched a custom enterprise agent to a client thinking it would be maybe 50/50 time on data vs agent tuning, but it's more like 99/1.

The alignment process goes very quickly once you have all the fish in exactly one barrel. I think pulling data dynamically from the source systems is where this turns into a game of whack-a-mole.

The problem with dynamic fetch is that you don't get any kind of persistent or compounding gains. There are queries that you simply cannot run because you'd chew through your GitHub, et. al., API quotas. It takes over 48h to fully hydrate the database for GitHub items on my current project. But, once that process is complete I can query across things like issue comments and do crosscutting joins with the state of other vendor systems in milliseconds.

I am finding the MSSQL dialect to be quite agreeable to the OAI models. With absolutely no prompting they will bootstrap off information schema and extended description properties every single time. If you design the schema for your audience, the amount of "Jesus prompting" you will require is much better controlled.

Re: Building reliable agentic AI systems

#12

Post by ares623 »

altmanaltman wrote: post #9

> The author used AI assistance during the writing of this article. AI tools were used for brainstorming ideas, creating outlines, and reviewing drafts to polish language and improve clarity. The first sentence makes it seem like they just used to improve sentence structure etc but the second line makes it seem like they used it for 90% of the work. Which one is true?

Your question answered it I think. The first sentence aims to mislead. The second sentence covers their ass.

I'd love to see the number of man hours that led to that sentence, and how proud they were to have come up with it.

Re: Building reliable agentic AI systems

#13

Post by niyikiza »

padolsey wrote: post #5

These vast multi-agentic systems with roles like 'Researcher', 'Writer' (with a review loop), 'Reflection agent', seem to ~feel~ mostly right but lack evals as to the merit of agent decomposition. So it forms a satisfying enough flowchart but I see no evidence these authors actually tried other approaches or agent roles. And let's be honest: an agent is just a system prompt and output contracts, and these rich archit…

What would the benefit be? A mega agent that does everything?

There are some well documented advantages of decomposition...that's why the industry favours microservices over monoloths.

Re: Building reliable agentic AI systems

#14

Post by dbuxton »

bob1029 wrote: post #11

The most important part is the database that the agent can see and how clean the data is. I pitched a custom enterprise agent to a client thinking it would be maybe 50/50 time on data vs agent tuning, but it's more like 99/1. The alignment process goes very quickly once you have all the fish in exactly one barrel. I think pulling data dynamically from the source systems is where this turns into a game of whack-a-mole…

The problem with this is that you have a consistency problem if you want to take action. The only way of making your agents read-write rather than read-only in practice is to use the underlying systems rather than try and pool information in a data lake.

But that does make it more complex to build simple information retrieval use cases.

Re: Building reliable agentic AI systems

#15

Post by Tanxsinxlnx »

bob1029 wrote: post #11

The most important part is the database that the agent can see and how clean the data is. I pitched a custom enterprise agent to a client thinking it would be maybe 50/50 time on data vs agent tuning, but it's more like 99/1. The alignment process goes very quickly once you have all the fish in exactly one barrel. I think pulling data dynamically from the source systems is where this turns into a game of whack-a-mole…

[flagged]

Re: Building reliable agentic AI systems

#16

Post by bob1029 »

dbuxton wrote: post #14
bob1029 wrote: post #11

The most important part is the database that the agent can see and how clean the data is. I pitched a custom enterprise agent to a client thinking it would be maybe 50/50 time on data vs agent tuning, but it's more like 99/1. The alignment process goes very quickly once you have all the fish in exactly one barrel. I think pulling data dynamically from the source systems is where this turns into a game of whack-a-mole…

The problem with this is that you have a consistency problem if you want to take action. The only way of making your agents read-write rather than read-only in practice is to use the underlying systems rather than try and pool information in a data lake. But that does make it more complex to build simple information retrieval use cases.

You definitely need multiple paths. It's not one vs the other. The data warehouse is immutable to the agent. There are separate tools that handle any updates to business state.

Re: Building reliable agentic AI systems

#17

Post by ThePhysicist »

I think for mostly search-focused use case like the one presented here AI is great as you don't ask it to build stuff or invent new drugs, you just want to retrieve relevant documents with laser precision, and agents can do that.

I think right now I'm mostly disappointed with agents writing code as they always degrade the quality of the codebase after a while, and the same goes for writing in general which just requires a ton of editing and mostly just sounds good but doesn't have a lot of substance in the end. I think you can really tell that these systems are trained to just produce plausible streams of text, especially in longer artefacts you notice that locally the inner consistency of what they produce is great but globally it really falls apart, it's like seeing the limits of their "intelligence".

For search however I really like AI, it has improved information retrieval so much for me where before I had to think about which keywords to use and combine and which filters to apply, describing what I'm looking for in plain text and then having the AI find it for me feels magical. Recently I wanted to find an artist that I heard in some old episode of the KEXP runcast (a running podcast), and I didn't remember anything except that it was rap with a kind of monotone voice a fast beat and a strong accent. Googles' agent asked a few clarifying questions and after a few rounds it found the artist for me, Genesis Uwusu. That's why I think Google will win in the AI assisted search market, they just have the best integration between fast and reasonably "smart" agents and high quality search data. Claude or ChatGPT are too slow and don't have fast enough data retrieval it seems, using them for search feels quite sluggish in comparison.

Re: Building reliable agentic AI systems

#19

Post by cpinto »

dbuxton wrote: post #14
bob1029 wrote: post #11

The most important part is the database that the agent can see and how clean the data is. I pitched a custom enterprise agent to a client thinking it would be maybe 50/50 time on data vs agent tuning, but it's more like 99/1. The alignment process goes very quickly once you have all the fish in exactly one barrel. I think pulling data dynamically from the source systems is where this turns into a game of whack-a-mole…

The problem with this is that you have a consistency problem if you want to take action. The only way of making your agents read-write rather than read-only in practice is to use the underlying systems rather than try and pool information in a data lake. But that does make it more complex to build simple information retrieval use cases.

I’m trying to solve this problem and would love to get some feedback: https://entidade.wls-labs.com

Re: Building reliable agentic AI systems

#20

Post by flir »

padolsey wrote: post #5

These vast multi-agentic systems with roles like 'Researcher', 'Writer' (with a review loop), 'Reflection agent', seem to ~feel~ mostly right but lack evals as to the merit of agent decomposition. So it forms a satisfying enough flowchart but I see no evidence these authors actually tried other approaches or agent roles. And let's be honest: an agent is just a system prompt and output contracts, and these rich archit…

I've been wondering lately if old-school cybernetics might help there.

But my off-the-cuff, uninformed opinion is that the precise structure doesn't matter too much, and the impact these structures really have is that they allow More Tokens for the Token Furnace.

Post reply on HN