Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We replaced H.264 streaming with JPEG screenshots (and it worked better)

blog.helix.ml

Post reply on HN
Print view View original

231–240 of 332 posts

Re: We replaced H.264 streaming with JPEG screenshots (and it worked better)

#231

Post by nazgul17 »

mikepavone wrote: post #102

> When the network is bad, you get... fewer JPEGs. That’s it. The ones that arrive are perfect. This would make sense... if they were using UDP, but they are using TCP. All the JPEGs they send will get there eventually (unless the connection drops). JPEG does not fix your buffering and congestion control problems. What presumably happened here is the way they implemented their JPEG screenshots, they have some mechani…

Regarding the encoding efficiency, I imagine the problem is that the compromise in quality shows in the space dimension (aka fewer or blurry pixels) rather than in time. Users need to read text clearly, so the compromise in the time dimension (fewer frames) sounds just fine.

Re: We replaced H.264 streaming with JPEG screenshots (and it worked better)

#233

Post by cyberrock »

HelloUsername wrote: post #40
Dylan16807 wrote: post #15

> When the network is bad, you get... fewer JPEGs. That’s it. The ones that arrive are perfect. You can have still have weird broken stallouts though. I dunno, this article has some good problem solving but the biggest and mostly untouched issue is that they set the minimum h.264 bandwidth too high. H.264 can do a lot better than JPEG with a lot less bandwidth. But if you lock it at 40Mbps of course it's flaky. Try 1…

> Try 1Mbps and iterate from there. From the article: “Just lower the bitrate,” you say. Great idea. Now it’s 10Mbps of blocky garbage that’s still 30 seconds behind.

10Mbits is more than the maximum ingest bitrate allowed on Twitch. Granted, anyone who watches a recent game or an IRL stream there might tell you that it should go up to 12 or 15, but I don't think an LLM interface should have trouble. This feels like someone on a 4K monitor defeating themselves through their hedonic treadmill.

Re: We replaced H.264 streaming with JPEG screenshots (and it worked better)

#234

Post by Dylan16807 »

lewq wrote: post #221

Hi, author of the post here. Just fixed up some formatting issues from when we copied it into substack, sorry about that. Yeah, I used Opus 4.5 to help me write it (and it actually made me laugh!). But the struggle was real. Something I didn't make clear enough in the post is that jpeg works because each screenshot is taken exactly when it's requested. Whereas streaming video is pushing a certain frame rate. The clie…

> Something I didn't make clear enough in the post is that jpeg works because each screenshot is taken exactly when it's requested. Whereas streaming video is pushing a certain frame rate. The client driving the frame rate is exactly what makes it not queue frames.

I understand that logic but I don't really agree with it. Very aggressive bitrate controls can do a lot to keep that buffer tiny while still looking better than JPEG, and if it bloats beyond 1-2 seconds you can reset. A reset like that wouldn't look notably worse than JPEG mode always looks.

If you use a video encoder that gives you good insight into what it's doing you could guarantee that the buffer never gets bigger than 1-2 JPEGs by dynamically deciding when to add frames. That would give you the huge benefits of P-frames with no downside.

Re: We replaced H.264 streaming with JPEG screenshots (and it worked better)

#235

Post by phire »

martinald wrote: post #206
HelloUsername wrote: post #40

Earlier quoted context omitted.

> Try 1Mbps and iterate from there. From the article: “Just lower the bitrate,” you say. Great idea. Now it’s 10Mbps of blocky garbage that’s still 30 seconds behind.

The problem is I think that they are using moonlight which is "designed" to stream games at very low latency. I very much doubt that people need When you try and use h264 et al at low latency you have to get rid of a lot of optimisations to encode it as quickly as possible. I also highly suspect the vaapi encoder is not very good esp at low bitrates. I _think_ moonlight also forces CBR instead of VBR, which is pretty…

Yeah, i think the author has been caught out by the fact that there simply isn’t a canonical way to encode h264.

JPEG is nice and simple, most encoders will produce (more or less) the same result for any given quality settings. The standard tells you exactly how to compress the image. Some encoders (like mozjpeg) use a few non-standard tricks to produce 5-20% better compression, but it’s essentially just a clever lossy preprocessing pass.

With h264, the standard essentially just says how decompressors should work, and it’s up to the individual encoders to work out to make best use of the available functionality for their intended use case. I’m not sure any encoder uses the full functionality (x264 refuses to use arbitrary frame order without b-frames, and I haven’t found an encoder that takes advantage of that). Which means the output of different encoders has wildly different results.

I’m guessing moonlight makes the assumption that most of its compression will come from motion prediction, and then takes massive shortcuts when encoding iframes.

Re: We replaced H.264 streaming with JPEG screenshots (and it worked better)

#236

Post by bugufu8f83 »

piskov wrote: post #166
bugufu8f83 wrote: post #165

Earlier quoted context omitted.

>Setting aside...the LLM writing style I don't want to set that aside either. Why is AI generated slop getting voted to the top of HN? If you can't be bothered to spend the time writing a blog post, why should I be bothered spending my time reading it? It's frankly a little bit insulting.

Don’t assume something you cannot prove. It was great writing

>Don’t assume something you cannot prove.

Well it's an inherently unprovable accusation, so assumption will have to do. It reeks of LLM-ese in certain word choices, phrases, and structure, though. I thought it was quite clear.

>It was great writing

Err... no accounting for taste, I suppose.

Re: We replaced H.264 streaming with JPEG screenshots (and it worked better)

#237

Post by kixelated »

lewq wrote: post #221

Hi, author of the post here. Just fixed up some formatting issues from when we copied it into substack, sorry about that. Yeah, I used Opus 4.5 to help me write it (and it actually made me laugh!). But the struggle was real. Something I didn't make clear enough in the post is that jpeg works because each screenshot is taken exactly when it's requested. Whereas streaming video is pushing a certain frame rate. The clie…

Hey lewq, 40Mbps is an absolutely ridiculous bitrate. For context, Twitch maxes out around 8.5Mb/s for 1440p60. Your encoder was poorly configured, that's it. Also, it sounds like your mostly static content would greatly benefit from VBR; you could get the bitrate down to 1Mb/s or something for screen sharing.

And yeah, the usual approach is to adapt your bitrate to network conditions, but it's also common to modify the frame rate. There's actually no requirement for a fixed frame rate with video codecs. It also you could do the same "encode on demand" approach with a codec like H.264, provided you're okay with it being low FPS on high RTT connections (poor Australians).

Overall, using keyframes only is a very bad idea. It's how the low quality animated GIFs used to work before they were secretly replaced with video files. Video codecs are extremely efficient because of delta encoding.

But I totally agree with ditching WebRTC. WebSockets + WebCodecs is fine provided you have a plan for bufferbloat (ex. adaptive bitrate, ABR, GoP skipping).

Re: We replaced H.264 streaming with JPEG screenshots (and it worked better)

#239

Post by mikepavone »

nazgul17 wrote: post #231
mikepavone wrote: post #102

> When the network is bad, you get... fewer JPEGs. That’s it. The ones that arrive are perfect. This would make sense... if they were using UDP, but they are using TCP. All the JPEGs they send will get there eventually (unless the connection drops). JPEG does not fix your buffering and congestion control problems. What presumably happened here is the way they implemented their JPEG screenshots, they have some mechani…

Regarding the encoding efficiency, I imagine the problem is that the compromise in quality shows in the space dimension (aka fewer or blurry pixels) rather than in time. Users need to read text clearly, so the compromise in the time dimension (fewer frames) sounds just fine.

Nothing stopping you from encoding h264 at a low frame rate like 5 or 10 fps. In webRTC, you can actually specify how you want to handle low bitrate situations with degredationPreference. If set to maintain-resolution, it will prefer sacrificing frame rate.
Post reply on HN