Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building reliable agentic AI systems

martinfowler.com

Post reply on HN
Print view View original

31–40 of 74 posts

Re: Building reliable agentic AI systems

#31

Post by CuriouslyC »

bingemaker wrote: post #25
bob1029 wrote: post #11

The most important part is the database that the agent can see and how clean the data is. I pitched a custom enterprise agent to a client thinking it would be maybe 50/50 time on data vs agent tuning, but it's more like 99/1. The alignment process goes very quickly once you have all the fish in exactly one barrel. I think pulling data dynamically from the source systems is where this turns into a game of whack-a-mole…

I have a question: How does connecting agent to db directly work in case of multi tenant system? There is a high chance that agent can snoop into multiple tenants and mess up the responses

With postgres you can use schemas to keep tenants separate and use RLS on shared data.

Re: Building reliable agentic AI systems

#32

Post by hilariously »

bob1029 wrote: post #11

The most important part is the database that the agent can see and how clean the data is. I pitched a custom enterprise agent to a client thinking it would be maybe 50/50 time on data vs agent tuning, but it's more like 99/1. The alignment process goes very quickly once you have all the fish in exactly one barrel. I think pulling data dynamically from the source systems is where this turns into a game of whack-a-mole…

That's funny because I have worked with hundreds and hundreds of TSQL products and only those built mostly in the last century have extended attributes (documenting each object in the database itself) - its nice but clunky and few things support it.

Re: Building reliable agentic AI systems

#35

Post by BodyCulture »

Most important piece of information is in the linked Frontiers article:

   However, the overall capability of the chatbot to fully meet user needs received a lower average score (3.1/5.0), highlighting the need for further improvements.
Also there is still the problem of hallucinations, as we see in the „Evaluation“ paragraph:

   Live traffic evaluations are essential for monitoring system behavior, identifying potential issues like hallucinations in production, and understanding performance on diverse live queries.
This are quite devastating results. This is a system for scientific research on medicines and mediocrity and hallucinations will kill people.

Would be interesting to know how much money was flushed down the toilet with these experts.

Re: Building reliable agentic AI systems

#36

Post by bob1029 »

bingemaker wrote: post #25
bob1029 wrote: post #11

The most important part is the database that the agent can see and how clean the data is. I pitched a custom enterprise agent to a client thinking it would be maybe 50/50 time on data vs agent tuning, but it's more like 99/1. The alignment process goes very quickly once you have all the fish in exactly one barrel. I think pulling data dynamically from the source systems is where this turns into a game of whack-a-mole…

I have a question: How does connecting agent to db directly work in case of multi tenant system? There is a high chance that agent can snoop into multiple tenants and mess up the responses

I think this mostly depends on your business model.

In my client's business, the idea of having all their customer knowledge contained in one global scope is a fantasy, not a fear.

I suppose if you were granting access to users outside the business that this could become a concern, but I haven't encountered anyone who is interested in that yet.

Re: Building reliable agentic AI systems

#37

Post by amw-zero »

stevex wrote: post #30

You can almost tell the "era" that a solution was built in these days since things are changing so fast. Mid-2026, we have very large context windows, and much smarter models than we did in 2024 when this was built. If I were to tackle this today I'd ask a current frontier model to work through the source data and design a hierarchy that would give it the ability to sift through the content itself by drilling down as…

It would not, and you would know that if you actually evaluated the results.

Re: Building reliable agentic AI systems

#38

Post by agentdev001 »

amw-zero wrote: post #37
stevex wrote: post #30

You can almost tell the "era" that a solution was built in these days since things are changing so fast. Mid-2026, we have very large context windows, and much smarter models than we did in 2024 when this was built. If I were to tackle this today I'd ask a current frontier model to work through the source data and design a hierarchy that would give it the ability to sift through the content itself by drilling down as…

It would not, and you would know that if you actually evaluated the results.

I have gone through this process and evaluated the results. Maybe you're referring to their comment as written, but going through what OC described + handholding leads to very good results in my experience.

Re: Building reliable agentic AI systems

#39

Post by agentdev001 »

I find papers/articles which discuss solutions that rely heavily on a model in the middle unreadable, if the models used are not discussed.

The data you need to get into context for a small model, vs a big boy frontier model, vs a fine tuned open weight big boy- are all very different. I can understand what they're doing here, and most of the 'why', but- not all of the why.

Re: Building reliable agentic AI systems

#40

Post by dominotw »

stevex wrote: post #30

You can almost tell the "era" that a solution was built in these days since things are changing so fast. Mid-2026, we have very large context windows, and much smarter models than we did in 2024 when this was built. If I were to tackle this today I'd ask a current frontier model to work through the source data and design a hierarchy that would give it the ability to sift through the content itself by drilling down as…

i can almost tell then you have not done anything like this in production scale. context window size is irrelevant.
Post reply on HN