Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Examples of agentic LLM systems in production?

news.ycombinator.com

Post reply on HN
Print view View original

51–60 of 80 posts

Re: Ask HN: Examples of agentic LLM systems in production?

#51

Post by JTyQZSnP3cQGa8B »

beernet wrote: post #48
isoprophlex wrote: post #29

An anecote that helps you maybe: I do contracting work, we're building a text-to-sql automated business analyst. It's quite well-rounded: it tries to recover from errors, allows automatic creation of appropriate visualisations, has a generic "faq" component to help the user understand how to use the tool. The tool is available to some 10.000 b2b users. It's just a bunch of prompts conditionally slapped together in a…

Am I the only one who finds these types of comments arrogant? I mean, we get it, you know better and have been doing this for a long time and so forth...Sometimes I feel like it's just about relativizing whatever tech is popular right now. Just to come back two years later and say "oh well I've been telling people about this cool tech two years ago!"

Give a counter example then. I’ve been doing this for years: people want the hot new thing even if it’s the worst idea, you rebrand it, and everyone is happy. Then a few months later, people praise you for not having implemented that bad idea.

Re: Ask HN: Examples of agentic LLM systems in production?

#53

Post by mistrial9 »

simonw wrote: post #33
mistrial9 wrote: post #28

Earlier quoted context omitted.

no "Agents" have a specific technical meaning.. an engine is connected to tools.. simple example is a bash terminal environment.

Where did you get the idea that "LLM connected to tools" is the one true meaning of the term agents? (I'm not trying to be accusatory here, just trying to understand how these opinions spread.)

Dawn Song final lecture at LLM class, UC Berkeley two weeks ago

https://understanding-ai-safety.org/

Re: Ask HN: Examples of agentic LLM systems in production?

#54

Post by zdifferentiator »

Yesterday I recorded an example of an O'reilly auto parts customer service agent to show how users can invoke them using RAG - last part of this video https://youtu.be/Qk_pVHtgcyA

There are plenty of RAG-capable LLMs in production, but still few products/UX oriented toward agentic work.

An AI product that can make purchases and API requests to external services like delivery drivers, calendars, etc. is still needed to truly enable these "agents" - which right now are basically read-only domain-specific LLMs.

Re: Ask HN: Examples of agentic LLM systems in production?

#55

Post by neilellis »

nisten wrote: post #32

They're ALL bullshit and there's a technical reason why. Your rube goldberg contraption that you put together for your borderline-fradulent pitch deck is NOT an assembly-line nor is it a product anyone's gonna buy. Why? Because cosine similarity search mathematically sucks a* , large context windows, while better, are nowhere close to being fast and practical ( maybe with a small exception of the generic sounding 1M…

Not sure why that was downvoted TBH, seems accurate to me.

Re: Ask HN: Examples of agentic LLM systems in production?

#57

Post by philipodonnell »

isoprophlex wrote: post #29

An anecote that helps you maybe: I do contracting work, we're building a text-to-sql automated business analyst. It's quite well-rounded: it tries to recover from errors, allows automatic creation of appropriate visualisations, has a generic "faq" component to help the user understand how to use the tool. The tool is available to some 10.000 b2b users. It's just a bunch of prompts conditionally slapped together in a…

I’ve been doing a lot of work on semantic data architecture that better supports LLM analytics, did you use any framework or methodology to decide how exactly to present the data/metadata to the LLM context to allow it to make decisions?

Re: Ask HN: Examples of agentic LLM systems in production?

#58

Post by t55 »

We built several LLM-powered applications that collectively served thousands of users. The biggest challenge we faced was ensuring reliability: making sure the workflows were robust enough to handle edge cases and deliver consistent results.

In practice, achieving this reliability meant repeatedly:

1. Breaking down complex goals into simpler steps: Composing prompts, tool calls, parsing steps, and branching logic. 2. Debugging failures: Identifying which part of the workflow broke and why. 3. Measuring performance: Assessing changes against real metrics to confirm actual improvement.

We tried some existing observability tools or agent frameworks and they fell short on at least one of these three dimensions. So we built our own: https://github.com/PySpur-Dev/PySpur

1. Graph-based interface: We can lay out an LLM workflow as a node graph. A node can be an LLM call, a function call, a parsing step, or any logic component. The visual structure provides an instant overview, making complex workflows more intuitive. 2. Integrated debugging: When something fails, we can pinpoint the problematic node, tweak it, and re-run it on some test cases right in the UI. 3. Evaluate at the node level: We can assess how node changes affect performance downstream.

We hope it's useful for other LLM developers out there

Re: Ask HN: Examples of agentic LLM systems in production?

#59

Post by moyix »

We've been using them to find novel vulnerabilities in open source web apps. The past 4 posts here have details:

- Auth bypass/arbitrary file read in Scoold: https://xbow.com/blog/xbow-scoold-vuln/

- SSRF in 2FAuth: https://xbow.com/blog/xbow-2fauth-ssrf/

- Stored XSS in 2FAuth: https://xbow.com/blog/xbow-2fauth-xss/

- Path traversal in Labs.AI EDDI: https://xbow.com/blog/xbow-eddi-path/

Each of those has an associated agent trace so you can go read exactly what the agent did to find and exploit the vulnerability.

Re: Ask HN: Examples of agentic LLM systems in production?

#60

Post by zdifferentiator »

wpietri wrote: post #24
SebaSeba wrote: post #4

Earlier quoted context omitted.

When does a LLM customer support bot that is based for example on RAG architecture, become an LLM agent?

My take is that if the LLM outputs text for humans to read, that's not an agent. If it's making API calls and doing things with the results, that's an agent. But given the way "AI" has stretched to become the new "radium" [1], I'm sure "agent" will shortly become almost meaningless. [1] https://en.wikipedia.org/wiki/Radium_fad

^^ best definition.

Right now they are "read-only" which I would call a persona

Post reply on HN