Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We replaced H.264 streaming with JPEG screenshots (and it worked better)

blog.helix.ml

Post reply on HN
Print view View original

211–220 of 332 posts

Re: We replaced H.264 streaming with JPEG screenshots (and it worked better)

#211

Post by netsharc »

npunt wrote: post #175
piskov wrote: post #166

Earlier quoted context omitted.

Don’t assume something you cannot prove. It was great writing

Normally the 1 sentence per para LinkedIn post for dummies writing style bugs me to no end, but for a technical article that's continually hopping between questions, results, code, and explanations, it fits really well and was a very easy article to skim and understand.

It's action thriller writing for something that's in reality is super dull (my question is loaded with outdated cliches, but would you be telling a girl you're trying to impress at a party about this problem you faced of trying to push some data over the network?). I had to skim over it, like watching a YouTube video at 2x so I don't start evaluating how obnoxious the narrator is.

Re: We replaced H.264 streaming with JPEG screenshots (and it worked better)

#212

Post by dehrmann »

> What if we only send keyframes?

I think the author reached this conclusion, but individual jpegs is essentially only keyframes.

> We don’t spam HTTP requests for individual frames like it’s 2009.

Uncompressed frames are huge, somewhere between 5 MB and 50 MB. The overhead of a request is negligible. It's also different when you're optimizing for latency and reliability where dropped frames is OK. Really, the lesson is they should have tried the easy thing first to see how good it was.

Re: We replaced H.264 streaming with JPEG screenshots (and it worked better)

#213

Post by plqbfbv »

I dabbled a bit with recoding/encoding videos in the past: 40mbps is basically blu-ray quality (1080p/4k depending on content), and it's being used to stream a mostly-static background with some text scrolling in front of it.

A 3-minute chat with Claude suggests 30FPS should be plenty (perhaps minor cursor lag can be noticed if it's drawn), with a GOP of 2s (60 frames) for fast recovery, VBR 1mbps average with a max bitrate at 1.2mbps for crappy connections, and bframes to minimize bandwidth usage (because we have hw encoding).

The crappiest of internet cafes should still be able to guarantee 1.2mbps (150kb/s). If they can do 5-10FPS with 150kb frames, they have 6-12mbps available. Worst case GOP can be reduced to 15 frames, so that there's 2x I-frames every second, and the latency is 500ms tops.

Re: We replaced H.264 streaming with JPEG screenshots (and it worked better)

#214

Post by dehrmann »

> We’re building Helix, an AI platform where autonomous coding agents work in cloud sandboxes. Users need to watch their AI assistants work. Think “screen share, but the thing being shared is a robot writing code.”

This feels like a fast dead end. Agents will get much faster pretty quickly, so synchronous human supervision isn't going to scale. I'd focus on systems that make high-signal asks of humans asynchronously.

Re: We replaced H.264 streaming with JPEG screenshots (and it worked better)

#215

Post by [deleted] »

nathan82 wrote: post #205
rdsubhas wrote: post #191

Earlier quoted context omitted.

Huh? This is the least LLM writing style I've encountered. Extraordinary claims require extraordinary proof.

It's not an extraordinary claim, it's a mundane and plausible one. This is exactly what you get when you ask an LLM to write in a "engaging conversational" style, and skip any editing after the fact. You could never prove it but there are a LOT of tells. "The key insight" - llms love key insights! "self-contained corruption-free" - they also love over-hypenating, as much as they love em-dashing. Both abundant here. "…

[deleted]

Re: We replaced H.264 streaming with JPEG screenshots (and it worked better)

#216

Post by socketcluster »

Next phase would be to do diffs between the JPEGs and if the diff is smaller than the next JPEG, only send the (gzipped) diff and reconstruct the next JPEG on the client side.

TBH, the obsession with standards is kind of nutty. It's not that hard to implement custom solutions that are better adapted to specific problems. Standards make sense when you want maximum interoperability but not everything requires this degree of interoperability these days. It's not such hassle to just provide a lightweight client in those cases.

For example, it's not ideal to use HTTP2 server push for realtime chat use cases. It was primarily intended for file push to avoid round-trip latency but HTTP is such a powerful and widespread protocol that people feel the need to use it for everything.

Re: We replaced H.264 streaming with JPEG screenshots (and it worked better)

#217

Post by giantrobot »

somehnguy wrote: post #172

40mbps for video of an LLM typing text didn't immediately fire off alarm bells in anyone's head that their approach was horribly wrong? That's an insane amount of bandwidth for what they're trying to do.

And they for some reason need a 60fps stream to...watch a computer type. No one stopped for a second and asked "maybe we don't know anything about the problem domain". They seem to have given a vague description to an LLM and assumed it knew what it was talking about.

Re: We replaced H.264 streaming with JPEG screenshots (and it worked better)

#218

Post by boggyb »

You can make webrtc work on enterprise networks by tunneling turn tcp traffic over websocket. The flow looks like this.

client's webrtc app using turn (pointing to the same machine IP) tcp server/ websocket client (runs on client machine) websocket server (relays turn packets) real turn server host's webrtc app

https://github.com/amitv87/turn_ws_proxy

I implemented a similar technique for Browserstack more than a decade ago to bypass enterprise firewalls by tunneling turn packets over (websockets/sse/socket.io etc.) The `tcp server/ websocket/sse/scoket.io client` was hosted as part of a packaged chrome app / firefox extension. WebSocket and TURN servers were hosted on same machine to minimize the latency (could have been embedded in same process to reduce latency further).

Re: We replaced H.264 streaming with JPEG screenshots (and it worked better)

#219

Post by kimixa »

TacticalCoder wrote: post #115
MBCook wrote: post #65

So it’s video of an AI typing text? Why not just send text? Why do you need video at all?

You apparently need video for the 45 seconds window you then get before preventing catastrophic things to happen. From TFA: > You’re watching the AI type code from 45 seconds ago > > By the time you see a bug, the AI has already committed it to main > > Everything is terrible forever Is this satire? I mean: if the solution for things to not be terrible forever consists in catching what an AI is doing in 45 seconds (!…

If you can realistically notice and reason out a bug within ~45 seconds of seeing the diff, then they are really shallow "dumb" bugs. The sort that even a junior would be expected to avoid.

And I wonder how many other massive issues are being committed to main, but would take longer to reason out, but you're already looking at the next 45-second shallow bug.

This has to be a joke, right?

Re: We replaced H.264 streaming with JPEG screenshots (and it worked better)

#220

Post by 6r17 »

londons_explore wrote: post #167
mikepavone wrote: post #160

Earlier quoted context omitted.

> They shared the polling code in the article. It doesn't request another jpeg until the previous one finishes downloading. You're right, I don't know how I managed to skip over that. > UDP is not necessary to write a loop. True, but this doesn't really have anything to do with using JPEG either. They basically implemented a primitive form of rate control by only allowing a single frame to be in flight at once. It wa…

> have limited control over their encode pipeline. Frustratingly this seems common in many video encoding technologies. The code is opaque, often has special kernel, GPU and hardware interfaces which are often closed source, and by the time you get to the user API (native or browser) it seems all knobs have been abstracted away and simple things like choosing which frame to use as a keyframe are impossible to do. I h…

I wonder if we could scan / test / dig these hidden features somehow ; like in a scrapping / fuzz fashion
Post reply on HN