Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We replaced H.264 streaming with JPEG screenshots (and it worked better)

blog.helix.ml

Post reply on HN
Print view View original

261–270 of 332 posts

Re: We replaced H.264 streaming with JPEG screenshots (and it worked better)

#261

Post by lelanthran »

mikepavone wrote: post #102

> When the network is bad, you get... fewer JPEGs. That’s it. The ones that arrive are perfect. This would make sense... if they were using UDP, but they are using TCP. All the JPEGs they send will get there eventually (unless the connection drops). JPEG does not fix your buffering and congestion control problems. What presumably happened here is the way they implemented their JPEG screenshots, they have some mechani…

> Still, you can do an initial bandwidth probe and then look for increasing transmission latency as a sign that the network is congested. Back off your bitrate (and if needed reduce frame rate to maintain sufficient quality) until transmission latency starts to decrease again.

They said playing around with bitrate didn't reduce the latency; all that happened was they got blocky videos with the latency remaining the same.

Re: We replaced H.264 streaming with JPEG screenshots (and it worked better)

#262

Post by lelanthran »

tsimionescu wrote: post #250
londons_explore wrote: post #179

Earlier quoted context omitted.

A word processor can save it's state at an arbitrary point... That's what the save button is for, and it's functional at any point in the document writing process! In fact, nearly everything in computing is serializable - or if it isn't, there is some other project with a similar purpose which is. However this is not the case with video codecs - but this is just one of many examples of where the video codec landscape…

> A word processor can save it's state at an arbitrary point... That's what the save button is for, and it's functional at any point in the document writing process! No, they generally can't save their whole internal state to be resumed later, and definitely not in the document you were editing. For example, when you save a document in vim it doesn't store the mode you were in, or the keyboard macro step that was exe…

> No, they generally can't save their whole internal state to be resumed later, and definitely not in the document you were editing.

I broadly agree, but I feel you chose a poor example - Vim.

> For example, when you save a document in vim it doesn't store the mode you were in,

Without user-mods, it does in fact start up in the mode that you were in when you saved, because you can only save in command/normal mode.

> or the keyboard macro step that was executing,

Without user-mods, you aren't able to interrupt a macro that is executing anyway, so if you cannot save mid-macro, why would you load mid-macro?

> or the search buffer,

Vim, by default, "remembers" all my previous searches, all the macros, and all my undos, even across sessions. The undo history is remembered per file.

Re: We replaced H.264 streaming with JPEG screenshots (and it worked better)

#263

Post by any1 »

adamjs wrote: post #63

They might want to check out what VNC has been doing since 1998– keep the client-pull model, break the framebuffer up into tiles and, when client requests an update, perform a diff against last frame sent, composite the updated tiles client-side. (This is what VNC falls back to when it doesn’t have damage-tracking from the OS compositor) This would really cut down on the bandwidth of static coding terminals where 90%…

Yes, in fact, the protocol states that the client can queue up multiple requests. The purpose of this is to fill up the gap created by the RTT. It is actually quite elegant in its simplicity.

An extension was introduced for continuous updates that allows the server to push frames without receiving requests, so this isn't universally true for all RFB (VNC) software. This is implemented in TigerVNC and noVNC to name a few.

Of course, continuous updates have the buffer-bloat problem that we're all discussing, so they also implemented fairly complex congestion control on top of the whole thing.

Effectively, they just moved the role of congestion control over to the server from the client while making things slightly more complicated.

Re: We replaced H.264 streaming with JPEG screenshots (and it worked better)

#264

Post by lomase »

somehnguy wrote: post #172

40mbps for video of an LLM typing text didn't immediately fire off alarm bells in anyone's head that their approach was horribly wrong? That's an insane amount of bandwidth for what they're trying to do.

That is where LLM shine. It lets you know who is a fraud.

40mbps to stream a terminal? Are you kidding me?

Re: We replaced H.264 streaming with JPEG screenshots (and it worked better)

#265

Post by josephernest »

antisol wrote: post #249
josephernest wrote: post #242

Related: for some hardware project, I have a backend server (either C++ or python) receiving frames from an industrial camera, uncompressed. And I need these frames displayed in a web browser client but on the same computer (instead of network trip like in this article). How would you do this ? I eventually did more or less like OP with uncompressed frames. My goal is to minimize CPU usage on the computer. Would h264…

> How would you do this ? It depends. I have many questions. > My goal is to minimize CPU usage on the computer. Would h264 compression be a good thing here given source and destination are the same machine? No. > Other ideas? 1. Why does it need to be displayed in a web browser (as opposed to more appropriate / better performing software specifically built for video)? 2. via what interface/library is the camera conn…

Thanks.

1. It is part of a bigger web-browser dashboard/control interface and this camera display is just one component among many others.

2. Some of the (USB) cameras can have proprietary interfaces such as https://www.ximea.com/support/wiki/apis/python

How would you do in this situation, to have the video stream in the browser, with as low CPU usage as possible?

3. Not for this project but for a future project, feel free to put a link to your portfolio or contact page (even if you remove the comment later)

Re: We replaced H.264 streaming with JPEG screenshots (and it worked better)

#266

Post by vlovich123 »

Sesse__ wrote: post #252
tsimionescu wrote: post #250

Earlier quoted context omitted.

> A word processor can save it's state at an arbitrary point... That's what the save button is for, and it's functional at any point in the document writing process! No, they generally can't save their whole internal state to be resumed later, and definitely not in the document you were editing. For example, when you save a document in vim it doesn't store the mode you were in, or the keyboard macro step that was exe…

> Serializable in principle, maybe. Actually serializable in the sense that the code contains a way to dump to a file and back, absolutely not. It's extremely rare for programs to expose a way to save and restore from a mid-state in the algorithm they're implementing. If you should ever look for an actual example; Cubemap, my video reflector ( https://manpages.debian.org/testing/cubemap/cubemap.1.en.htm... ), works l…

Why not hand off the fd to the new process spawned as a child? That’s how a lot of professional 0 downtime upgrades work: spawn a process, hand off fd & state, exit.

Re: We replaced H.264 streaming with JPEG screenshots (and it worked better)

#267

Post by zipy124 »

This is just poor engineering. H.264 streaming is obviously superior to JPEG streaming, else MJPEG (motion jpeg) would be standard for screen sharing. In addition if all you're sharing is a picture of text, and you have access to the text, you can just send the damn text instead and render it locally.

Re: We replaced H.264 streaming with JPEG screenshots (and it worked better)

#269

Post by antisol »

josephernest wrote: post #265
antisol wrote: post #249

Earlier quoted context omitted.

> How would you do this ? It depends. I have many questions. > My goal is to minimize CPU usage on the computer. Would h264 compression be a good thing here given source and destination are the same machine? No. > Other ideas? 1. Why does it need to be displayed in a web browser (as opposed to more appropriate / better performing software specifically built for video)? 2. via what interface/library is the camera conn…

Thanks. 1. It is part of a bigger web-browser dashboard/control interface and this camera display is just one component among many others. 2. Some of the (USB) cameras can have proprietary interfaces such as https://www.ximea.com/support/wiki/apis/python How would you do in this situation, to have the video stream in the browser, with as low CPU usage as possible? 3. Not for this project but for a future project, fee…

1. fair enough

2. "How would you do in this situation, to have the video stream in the browser, with as low CPU usage as possible?"

Since it's being consumed on (only) the local machine you've got an excellent situation where you can use any obscure codec you like, as long as the browser you're using supports it. Also you don't need to care at all about network bandwidth. If minimising CPU usage is the #1 priority then something fairly lightweight like mjpeg might do the trick. Alternatively you might get away with not compressing the video at all (but this might cause issues due to dealing with huge amounts of data). If I wanted to minimise CPU usage, I wouldn't be doing it in python.

3. You can find me if you look.

Post reply on HN