Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We replaced H.264 streaming with JPEG screenshots (and it worked better)

blog.helix.ml

Post reply on HN
Print view View original

171–180 of 332 posts

Re: We replaced H.264 streaming with JPEG screenshots (and it worked better)

#171

Post by colechristensen »

nemothekid wrote: post #168

I'm very familiar with the stack and the pain of trying to livestream video to a browser. If JPEG screenshots work for your clients, then I would just stick with that. The problem with wolf, gstreamer, moonlight, $third party, is you need to be familiar with how the underlying stack handles backpressure and error propagation, or else things will just "not work" and you will have no idea why. I've worked on 3 projects…

This is where LLMs shine, where you need to dip your toes into really complex systems but basically just to do one thing with pretty straightforward requirements.

Re: We replaced H.264 streaming with JPEG screenshots (and it worked better)

#173

Post by vikingerik »

StilesCrisis wrote: post #88
sevensor wrote: post #78

No mention of PNGs? I don’t usually go to jpegs first for screenshots of text. Did png have worse compression? Burn more cpu? I’m sure there are good reasons, but it seems like they’ve glossed over the obvious choice here. edit: Thanks for the answers! The consensus is that PNG en/de -coding is too expensive compared to jpeg.

PNGs are lossless so you can’t really dial up the compression. You can save space by reducing to 8-bit color (or grayscale!) but it’s basically the equivalent of raw pixels plus zlib.

PNG can be lossy. It can be done by first discarding some image detail, to make adjacent almost-matching pixel values actually match, to be more amenable to PNG's compression method. pngquant.org has a tool that does it.

There are usage cases where you might want lossy PNG over other formats; one is for still captures of 2d animated cartoon content, where H.264 tended to blur the sharp edges and flat color areas and this approach can compensate for that.

Re: We replaced H.264 streaming with JPEG screenshots (and it worked better)

#175

Post by npunt »

piskov wrote: post #166
bugufu8f83 wrote: post #165

Earlier quoted context omitted.

>Setting aside...the LLM writing style I don't want to set that aside either. Why is AI generated slop getting voted to the top of HN? If you can't be bothered to spend the time writing a blog post, why should I be bothered spending my time reading it? It's frankly a little bit insulting.

Don’t assume something you cannot prove. It was great writing

Normally the 1 sentence per para LinkedIn post for dummies writing style bugs me to no end, but for a technical article that's continually hopping between questions, results, code, and explanations, it fits really well and was a very easy article to skim and understand.

Re: We replaced H.264 streaming with JPEG screenshots (and it worked better)

#176

Post by Sesse__ »

londons_explore wrote: post #167
mikepavone wrote: post #160

Earlier quoted context omitted.

> They shared the polling code in the article. It doesn't request another jpeg until the previous one finishes downloading. You're right, I don't know how I managed to skip over that. > UDP is not necessary to write a loop. True, but this doesn't really have anything to do with using JPEG either. They basically implemented a primitive form of rate control by only allowing a single frame to be in flight at once. It wa…

> have limited control over their encode pipeline. Frustratingly this seems common in many video encoding technologies. The code is opaque, often has special kernel, GPU and hardware interfaces which are often closed source, and by the time you get to the user API (native or browser) it seems all knobs have been abstracted away and simple things like choosing which frame to use as a keyframe are impossible to do. I h…

> I couldn't find a single video codec which could do that without extensive internal surgery to save all internal state after the 15th frame.

fork()? :-)

But most software, video codec or not, simply isn't written to serialize its state at arbitrary points. Why would it?

Re: We replaced H.264 streaming with JPEG screenshots (and it worked better)

#178

Post by worksonmine »

TZubiri wrote: post #54
dotancohen wrote: post #51

They're just streaming a video feed of an LLC running in a terminal? Why not stream the actual text? Or fetch it piecemeal over AJAX requests? They complain that corporate networks support only HTTPS and nothing else? Do they not understand what the first T stands for?

Suppose an LLM opens a browser, or opens a corporate .exe and GUI and starts typing in there and clicking buttons.

You don't give it a browser or buttons to click.

Re: We replaced H.264 streaming with JPEG screenshots (and it worked better)

#179

Post by londons_explore »

Sesse__ wrote: post #176
londons_explore wrote: post #167

Earlier quoted context omitted.

> have limited control over their encode pipeline. Frustratingly this seems common in many video encoding technologies. The code is opaque, often has special kernel, GPU and hardware interfaces which are often closed source, and by the time you get to the user API (native or browser) it seems all knobs have been abstracted away and simple things like choosing which frame to use as a keyframe are impossible to do. I h…

> I couldn't find a single video codec which could do that without extensive internal surgery to save all internal state after the 15th frame. fork()? :-) But most software, video codec or not, simply isn't written to serialize its state at arbitrary points. Why would it?

A word processor can save it's state at an arbitrary point... That's what the save button is for, and it's functional at any point in the document writing process!

In fact, nearly everything in computing is serializable - or if it isn't, there is some other project with a similar purpose which is.

However this is not the case with video codecs - but this is just one of many examples of where the video codec landscape is limiting.

Another for example is that on the internet lots of videos have a 'poster frame' - often the first frame of the video. That frame for nearly all usecases ends up downloaded twice - once as a jpeg, and again inside the video content. There is no reasonable way to avoid that - but doing so would reduce the latency to play videos by quite a lot!

Re: We replaced H.264 streaming with JPEG screenshots (and it worked better)

#180

Post by thescriptkiddie »

egorfine wrote: post #5

> The constraint that ruined everything: It has to work on enterprise networks. > You know what enterprise networks love? HTTP. HTTPS. Port 443. That’s it. That’s the list. That's not enough. Corporate networks also love to MITM their own workstations and reinterpret http traffic. So, no WebSockets and no Server-Side Events either, because their corporate firewall is a piece of software no one in the world wants and…

> it likes to change the content while keeping Content-Length intact

thanks, i had repressed that memory

Post reply on HN