Trying to do something similar but using kasm[0] as the backend. [0] https://kasm.com
Fascinating, wanna compare notes on a call some time?
Streaming AI agent desktops with gaming protocols
41–50 of 52 posts
Re: Streaming AI agent desktops with gaming protocols
#42xpra has video streaming and allows for sharing
Interesting, thanks!
Re: Streaming AI agent desktops with gaming protocols
#43> because we’re all going to become managers of coding agents whether we like it or not I will join the woodworking people before that happens, thanks.
Either the crap truly works and nobody is needed or it does not work. Where is this half-arsed human-agent hybrid vision coming from? The land of plateaued LLM gains?
Yes
Re: Streaming AI agent desktops with gaming protocols
#44> because we’re all going to become managers of coding agents whether we like it or not I will join the woodworking people before that happens, thanks.
A career change that left me as a recent graduate in a decimated marketplace missing the bottom ten rungs on the ladder and no interest in getting back into the software world has led me to advanced manufacturing as a metal worker. I code a little, move heavy steel pieces periodically which is a nice way to break up the standing/sitting but not nearly as much as a general laborer, solve lots of problems, keep my trig…
I'm curious, hoping you might shed some insight.
I'm not in tech per se, but in education with a seasoning of tech.
Would you have any insight into what the field for education or training might look like for your new field?
I think (perhaps feel is better) that manufacturing will make some sort of comeback and that the gratification from it is actually quite beneficial for the human condition.
So, I'm interested in trying to glean what I might pivot into from the perspective of training and education.
Re: Streaming AI agent desktops with gaming protocols
#45I’m curious how far are we from giving coding agents access to these desktop agents so that when we are using say Claude Code to build a native desktop app, the coding agents can actually see and act on the desktop UI that it is building
For web apps, I'd guess that many of us already do that via Playwright or other MCPs. I'd bet there are people doing something similar with desktop apps too.
unfortunately there is nothing like that for the desktop apps.
Even mobile app devs have https://github.com/mobile-next
Re: Streaming AI agent desktops with gaming protocols
#46Re: Streaming AI agent desktops with gaming protocols
#47Earlier quoted context omitted.
For web apps, I'd guess that many of us already do that via Playwright or other MCPs. I'd bet there are people doing something similar with desktop apps too.
unfortunately there is nothing like that for the desktop apps. Even mobile app devs have https://github.com/mobile-next
Why not https://github.com/hrrrsn/mcp-vnc ?
Re: Streaming AI agent desktops with gaming protocols
#48I’m curious how far are we from giving coding agents access to these desktop agents so that when we are using say Claude Code to build a native desktop app, the coding agents can actually see and act on the desktop UI that it is building
The streaming architecture is designed for exactly this - we originally built it for autonomous agents that need persistent development environments. The missing pieces are mostly integration work (mapping Claude's tool use format to our desktop APIs). Would be very interested to hear if others are working on similar integrations - the combination of LLM coding agents + real desktop environments feels like it unlocks a lot of interesting workflows.
Re: Streaming AI agent desktops with gaming protocols
#49IMHO, the goals is not to have to watch what agents do and let them do the work. I would personally invest into making agents more autonomous (yes hard problem today) then building a desktop video session protocol to watch them do the work.
>I would personally invest into making agents more autonomous (yes hard problem today) then building a desktop video session protocol to watch them do the work. Seems difficult to research better autonomy without extensive monitoring. You need specific data on before/after effects of changes, for instance.
You're right. Monitoring is critical for this. You need to know exactly what changed and why.
This is actually where streaming + the actual architecture behing this becomes valuable beyond just "letting users watch." In Helix, every agent interaction is persisted with timing data (DurationMs), tool calls, error states, and even user feedback (thumbs up/down). Sessions track the full conversation history bidirectionally, so you can reconstruct exactly what the agent saw and did at each step.
Re: Streaming AI agent desktops with gaming protocols
#50This is beautiful madness.
Ha, we'll take it! It's definitely ambitious - we're essentially building a full remote desktop streaming stack on top of an AI stack on private data.
The 'madness' part is probably the part where we said 'let's do hardware-accelerated H.264 encoding and real-time agent control and make it all work with sub-100ms latency.' We think the problem space is worth the complexity.