Live data from Hacker News

Streaming AI agent desktops with gaming protocols

blog.helix.ml

21–30 of 52 posts

Re: Streaming AI agent desktops with gaming protocols

#21
Hi, quite interesting project but have a hard time to understand why would stream a desktop.

From my (ignorant) understanding, the important part is the context of the LLM in the task. Some conversations you need visuals, some you don't. What's the advantage of giving a full desktop streaming instead of using integrations?

Re: Streaming AI agent desktops with gaming protocols

#22

Hi, quite interesting project but have a hard time to understand why would stream a desktop. From my (ignorant) understanding, the important part is the context of the LLM in the task. Some conversations you need visuals, some you don't. What's the advantage of giving a full desktop streaming instead of using integrations?

Mainly so you can give the agent access to the desktop as well. Then it can debug your web app in Chrome Dev tools but also you can pair with it with streaming that is so good it feels local

Re: Streaming AI agent desktops with gaming protocols

#23

Hi, quite interesting project but have a hard time to understand why would stream a desktop. From my (ignorant) understanding, the important part is the context of the LLM in the task. Some conversations you need visuals, some you don't. What's the advantage of giving a full desktop streaming instead of using integrations?

There's also value in being able to run multiple agents in parallel with their own isolated filesystems and runtimes. One agent won't tread on the toes of another whatever they do. You can let it loose and it doesn't matter if it breaks something, you can just spin up another one

Re: Streaming AI agent desktops with gaming protocols

#25
post #24

I’m curious how far are we from giving coding agents access to these desktop agents so that when we are using say Claude Code to build a native desktop app, the coding agents can actually see and act on the desktop UI that it is building

This is a great point. Not that far. We also snapshot the desktop for "slow" non-streaming updates to the UI. We could push these into Claude itself to act on or describe or whatever.

Re: Streaming AI agent desktops with gaming protocols

#26
post #6

IMHO, the goals is not to have to watch what agents do and let them do the work. I would personally invest into making agents more autonomous (yes hard problem today) then building a desktop video session protocol to watch them do the work.

>I would personally invest into making agents more autonomous (yes hard problem today) then building a desktop video session protocol to watch them do the work.

Seems difficult to research better autonomy without extensive monitoring. You need specific data on before/after effects of changes, for instance.

Re: Streaming AI agent desktops with gaming protocols

#27
post #17

Earlier quoted context omitted.

A career change that left me as a recent graduate in a decimated marketplace missing the bottom ten rungs on the ladder and no interest in getting back into the software world has led me to advanced manufacturing as a metal worker. I code a little, move heavy steel pieces periodically which is a nice way to break up the standing/sitting but not nearly as much as a general laborer, solve lots of problems, keep my trig…

There’s 1,000 established industries that don’t offer the rapid growth and pay outs of the modern tech ecosystem. I’m excited to see some of the current industrial backwaters soak up technical talent freed up by the SV AI brain drain. To think we’ve handsomely paid our best and brightest the last few decades in pursuit of.. advertising?

> To think we’ve handsomely paid our best and brightest the last few decades in pursuit of.. advertising?

I think "efficiency" is more accurate there. Even post-Google/ad-tech-boom the overall trends that started decades earlier continued to be: (1) faster turnaround time on communications, (2) faster delivery of result artifacts, (3) faster knowledge of changes in the market and faster response.

Advertising is a particularly visible field with lots of money to throw at those things (active investment trading is another). But practically every other industry has chased those same things as well, all the way down to things like parking meters.

Personally I'm not convinced that this is such a great thing anyway - does anyone enjoy their boss messaging you at 11PM on any day they want whenever they get the fancy? - but that's the larger reason so much brainpower has been invested into it.

Re: Streaming AI agent desktops with gaming protocols

#29

> The Wolf maintainer has done heroic work ... I commend the fact they acknowledge the maintainer's work, but seeing the singular 'maintainer', I can't help but notice the weight on that one person's shoulders.

I should have said creator. He seems to have a healthy community backing him, but we should ask him!

Re: Streaming AI agent desktops with gaming protocols

#30
post #24

I’m curious how far are we from giving coding agents access to these desktop agents so that when we are using say Claude Code to build a native desktop app, the coding agents can actually see and act on the desktop UI that it is building

That's the next move :-D
Post reply on HN