Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We replaced H.264 streaming with JPEG screenshots (and it worked better)

blog.helix.ml

Post reply on HN
Print view View original

201–210 of 332 posts

Re: We replaced H.264 streaming with JPEG screenshots (and it worked better)

#201

Post by djmips »

tombert wrote: post #200
djmips wrote: post #106

Earlier quoted context omitted.

The blog post did smell of inexperience. Glad to hear there is other approaches - is something like that open source?

I'm not sure; sometimes being an experienced dev gravitates you towards the lazy solutions that are "good enough". Senior engineers are often expected to work at a rate that precludes solving interesting problems, and so the dumber solution will often win; at least that's been my experience, and what I tell myself to go to sleep at night when I get told for the millionth time that the company can't justify formal ver…

I understand what you're saying and certainly I've come up against that myself. I didn't intend my comment to be super pejorative.

Re: We replaced H.264 streaming with JPEG screenshots (and it worked better)

#202

Post by akersten »

didibus wrote: post #198

What I'm wondering is, why couldn't the AI generate this solution? And implement it all? Why did they need to spend human time and effort to experiment, arrive at this solution and implement it? I'm asking genuinely. I use GenAI a lot, every day, multiple times a day. It helps me write emails, documents, produce code, make configuration changes, create diagrams, research topics, etc. Still, it's all assisted, I never…

> What I'm wondering is, why couldn't the AI generate this solution? And implement it all?

My read of the blog post is that is exactly what happened, and the human time was mostly spent being confused why 40MB/s streams don't work well at a coffee shop.

Re: We replaced H.264 streaming with JPEG screenshots (and it worked better)

#203

Post by chrisweekly »

mikepavone wrote: post #102

> When the network is bad, you get... fewer JPEGs. That’s it. The ones that arrive are perfect. This would make sense... if they were using UDP, but they are using TCP. All the JPEGs they send will get there eventually (unless the connection drops). JPEG does not fix your buffering and congestion control problems. What presumably happened here is the way they implemented their JPEG screenshots, they have some mechani…

Related tangent: it's remarkable to me how a given jpeg can be literally visually indistinguishable from another (by a human on a decent monitor) yet consist of 10-15% as many bytes. I got pretty deep into web performance and image optimization in the late 2000s and it was gratifying to have so much low-hanging fruit.

Re: We replaced H.264 streaming with JPEG screenshots (and it worked better)

#204

Post by martinald »

ryukoposting wrote: post #142
adamjs wrote: post #63

They might want to check out what VNC has been doing since 1998– keep the client-pull model, break the framebuffer up into tiles and, when client requests an update, perform a diff against last frame sent, composite the updated tiles client-side. (This is what VNC falls back to when it doesn’t have damage-tracking from the OS compositor) This would really cut down on the bandwidth of static coding terminals where 90%…

Of all the suggestions in the comments here, this seems like the best one to start with. Also... I get that the dumb solution to "ugly text at low bitrates" is "make the bitrate higher." But still, nobody looked at a 40M minimum and wondered if they might be looking at this problem from the wrong angle entirely?

In fairness VNC-style approaches are bloody awful even over my 2.5gbit/sec lan on very fast hardware. It just cannot do 4K well (not sure if they need 4k or not).

I spent some time compiling the "new" xrdp with x264 and it is incredibly good, basically cannot really tell that I'm remote desktoping.

The bandwidth was extremely low as well. You are correct on that part, 40mbit/sec is nuts for high quality. I suspect if they are using moonlight it's optimized for extremely low latency at the expense of bandwidth?

Re: We replaced H.264 streaming with JPEG screenshots (and it worked better)

#205

Post by nathan82 »

rdsubhas wrote: post #191
qbow883 wrote: post #154

Setting aside the various formatting problems and the LLM writing style, this just seems all kinds of wrong throughout. > “Just lower the bitrate,” you say. Great idea. Now it’s 10Mbps of blocky garbage that’s still 30 seconds behind. 10Mbps should be way more than enough for a mostly static image with some scrolling text. (And 40Mbps are ridiculous.) This is very likely to be caused by bad encoding settings and/or a…

Huh? This is the least LLM writing style I've encountered. Extraordinary claims require extraordinary proof.

It's not an extraordinary claim, it's a mundane and plausible one. This is exactly what you get when you ask an LLM to write in a "engaging conversational" style, and skip any editing after the fact. You could never prove it but there are a LOT of tells.

"The key insight" - llms love key insights! "self-contained corruption-free" - they also love over-hypenating, as much as they love em-dashing. Both abundant here. "X like it's 2005" and also "Y like it's 2009" - what a cool casual turn of phrase, so natural! The architecture diagram is definitely unedited AI, Claude always messes up the border alignment on ascii boxes

I wouldn't mind except the end result is imprecise and sloppy, as pointed out by the GP comment. And the tone is so predictable/boring at this point, I'd MUCH rather read poorly written human output with some actual personality.

Re: We replaced H.264 streaming with JPEG screenshots (and it worked better)

#206

Post by martinald »

HelloUsername wrote: post #40
Dylan16807 wrote: post #15

> When the network is bad, you get... fewer JPEGs. That’s it. The ones that arrive are perfect. You can have still have weird broken stallouts though. I dunno, this article has some good problem solving but the biggest and mostly untouched issue is that they set the minimum h.264 bandwidth too high. H.264 can do a lot better than JPEG with a lot less bandwidth. But if you lock it at 40Mbps of course it's flaky. Try 1…

> Try 1Mbps and iterate from there. From the article: “Just lower the bitrate,” you say. Great idea. Now it’s 10Mbps of blocky garbage that’s still 30 seconds behind.

The problem is I think that they are using moonlight which is "designed" to stream games at very low latency. I very much doubt that people need When you try and use h264 et al at low latency you have to get rid of a lot of optimisations to encode it as quickly as possible. I also highly suspect the vaapi encoder is not very good esp at low bitrates.

I _think_ moonlight also forces CBR instead of VBR, which is pretty awful for this use case - imagine you have 9 seconds of 'nothing changing' and then the window moves for 0.25 seconds. If you had VBR the encoder could basically send ~0kbit/sec apart from control metadata, and then spike the bitrate up when the window moved (for brevity I'm simplifying here, it's more complicated than this but hopefully you get the idea).

Basically they've used the wrong software entirely. They should try and look at xrdp with x264 as a start.

Re: We replaced H.264 streaming with JPEG screenshots (and it worked better)

#207

Post by Tenobrus »

rdsubhas wrote: post #191
qbow883 wrote: post #154

Setting aside the various formatting problems and the LLM writing style, this just seems all kinds of wrong throughout. > “Just lower the bitrate,” you say. Great idea. Now it’s 10Mbps of blocky garbage that’s still 30 seconds behind. 10Mbps should be way more than enough for a mostly static image with some scrolling text. (And 40Mbps are ridiculous.) This is very likely to be caused by bad encoding settings and/or a…

Huh? This is the least LLM writing style I've encountered. Extraordinary claims require extraordinary proof.

ai detectors are never totally accurate but this one is quite good and it suggests something like 80% of this article is llm generated. honestly idk how you didn't get that just by reading it tho, maybe you haven't been exposed to much modern llm-generated content?

https://www.pangram.com/history/5cec2f02-6fd6-4c97-8e71-d509...

Re: We replaced H.264 streaming with JPEG screenshots (and it worked better)

#208

Post by j-me »

worksonmine wrote: post #178
TZubiri wrote: post #54

Earlier quoted context omitted.

Suppose an LLM opens a browser, or opens a corporate .exe and GUI and starts typing in there and clicking buttons.

You don't give it a browser or buttons to click.

I think we've passed the Rubicon when it comes to that

Re: We replaced H.264 streaming with JPEG screenshots (and it worked better)

#209

Post by saagarjha »

I’m currently doing this in one of my side projects: https://github.com/saagarjha/Ensemble. It works, kinda; it’s good enough for demos at least and I haven’t had much time to improve it. At some point you would really want to use an actual video encoder though because JPEGs are not cheap to encode and send even with hardware acceleration.
Post reply on HN