I think this is a good and pragmatic way to approach the use of LLM systems. By translating to an intermediate language, and then processing further symbolically. But probably you can be prompt injected also if you expose sensible "tools" to the LLM.
A2UI: A Protocol for Agent-Driven Interfaces
61–70 of 80 posts
Re: A2UI: A Protocol for Agent-Driven Interfaces
#62Earlier quoted context omitted.
A key challenge with HTML is client side trust. How do I enable an agent platform (say Gemini, Claude, OpenAI) to render UI from an untrusted 3p agent that’s integrated with the platform? This is a common scenario in the enterprise version of these apps - eg I want to use the agent from (insert saas vendor) alongside my company’s home grown agents and data. Most HTML is actually HTML+CSS+JS - IMO, accepting this is a…
Perhaps the protocol, is then html/css/js in a strict sandbox. Component has no access to anything outside of component bounds (no network, no dom/object access, no draw access, etc).
I think you can do that with an iframe, but it always makes me nervous
Re: A2UI: A Protocol for Agent-Driven Interfaces
#63> A2UI lets agents send declarative component descriptions that clients render using their own native widgets. It's like having agents speak a universal UI language . (emphasis mine) Sounds like agents are suddenly able to do what developers have failed at for decades: Writing platform-independent UIs. Maybe this works for simple use cases but beyond that I'm skeptical.
I've thought about how to write a platform independent UI framework that doesn't care what language you write it in, and every time I find myself reinventing X.org or at least my gut tells me I'm just reinventing a cross-platform X server implementation.
Re: A2UI: A Protocol for Agent-Driven Interfaces
#64So there's MCP-UI, OpenAI's ChatKit widgets and now Google's A2UI, that I know of. And probably some more... How many more variants are we introducing to solve the same problem. Sounds like a lot of wasted manhours to me.
AGUI sounds similar: https://github.com/ag-ui-protocol/ag-ui
This provides a bit more detail on how they relate to each other
Re: A2UI: A Protocol for Agent-Driven Interfaces
#65> A2UI lets agents send declarative component descriptions that clients render using their own native widgets. It's like having agents speak a universal UI language . (emphasis mine) Sounds like agents are suddenly able to do what developers have failed at for decades: Writing platform-independent UIs. Maybe this works for simple use cases but beyond that I'm skeptical.
Nope, it's just a repackaging of the same problem, except in this case, the problem is solved with APIs and CLI and not jumping through hoops in order to get the AI to do what humans do.
It's about accomplishing a task, not making a bot accomplish a task using the same tools and embodiment context as a human - there's no upside, unless the bot is actually using a humanoid embodiment, and even then, using a CLI and service API is going to be preferable to doing things with UI in nearly every possible case, except where you want to limit to human-ish capabilities, like with gaming, or you want to deceive any monitors into thinking that a human is operating.
It's going to be infinitely easier to wrap a json get/push wrapper around existing APIs or automation interfaces than to universalize some sort of GUI interactions, because LLM's don't have the realtime memory you need to adapt to all the edge cases on the fly. It's incredibly difficult for humans, and hundreds of billions of dollars have been spent trying to make software universally accessible and dumbed down for users, and still ends up being either stupidly limited, or fractally complex in the tail, and no developer can ever account for all the possible ways in which users interact with a feature for any moderately complex piece of software.
Just use existing automation patterns. This is one case where if an AI picks up this capability alongside other advances, then awesome, but any sort of middleware is going to be a huge hack that immediately gets obsoleted by frontier models as a matter of course.
Re: A2UI: A Protocol for Agent-Driven Interfaces
#66I wouldn't want this anywhere near production, but for rapid prototyping this seems great. People famously can't articulate what they want until they get to play around with it. This lets you skip right to the part where you realize they want something completely different from what was first described without having to build the first iteration by hand
Honestly the point of this is not to help app developers—it's to replace the need for apps altogether. The vision here is that you can chat with Gemini, and it can generate an app on the fly to solve your problem. For the visualized landscaping app, it could just connect to landscapers via their Google Business Profile. As an app developer, I'm actually not even against this. The amount of human effort that goes into…
This sounds like they creators think that even more duplicative apps that no one knows how it works or what the code even looks like... is a better idea?
How many times are users going to spin GPUs to create the same app?
Re: A2UI: A Protocol for Agent-Driven Interfaces
#67Am I reading (7) of the data flow correctly?
1. Establish SSE connection
... user event
7. send updates over origin SSE connection
So the client is required to maintain an SSE capable connection for the entire chat session? What if my network drops or I switch to another agent?
Seems an onerous requirement to maintain a connection for the life-time of a session, which can span days (as some people have told us they have done with agents)
Re: A2UI: A Protocol for Agent-Driven Interfaces
#68So there's MCP-UI, OpenAI's ChatKit widgets and now Google's A2UI, that I know of. And probably some more... How many more variants are we introducing to solve the same problem. Sounds like a lot of wasted manhours to me.
Unlike many of those approaches which concern themselves with delivery of human-designed static UI, this seems to be a tool designed to support generative UIs. I personally think that's a non-starter and much prefer the more incremental "let the agent call a tool that renders a specific pre-made UI" approach of MCP UI/Apps, OpenAI Apps SDK, etc for now.
Re: A2UI: A Protocol for Agent-Driven Interfaces
#69Earlier quoted context omitted.
If you play with A2UIs generator that's effectively what it does, just layer of abstraction or two above what you're describing.
That's what I thought too skimming through the documentation, my thinking is that since it does that, which makes sense to avoid script injection, why not do it with "jsonized" html.
I was thinking that raw html might be too verbose, but canned components have signatures and types.
Re: A2UI: A Protocol for Agent-Driven Interfaces
#70Earlier quoted context omitted.
Honestly the point of this is not to help app developers—it's to replace the need for apps altogether. The vision here is that you can chat with Gemini, and it can generate an app on the fly to solve your problem. For the visualized landscaping app, it could just connect to landscapers via their Google Business Profile. As an app developer, I'm actually not even against this. The amount of human effort that goes into…
This sounds like they creators think that even more duplicative apps that no one knows how it works or what the code even looks like... is a better idea? How many times are users going to spin GPUs to create the same app?
If Google's paying for the GPU time, I guess it's up to them how they want to cache apps for frequently-used queries. Glad I'm not paying for it!