Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We replaced H.264 streaming with JPEG screenshots (and it worked better)

blog.helix.ml

Post reply on HN
Print view View original

281–290 of 332 posts

Re: We replaced H.264 streaming with JPEG screenshots (and it worked better)

#281

Post by bArray »

I've literally been here (many) years ago whilst trying to stream video from a potato Linux SBC via WiFi. As you walked further away, the H264 stream would just die and hang, no matter what you did. Stream JPEGs? Worked excellently and adjusted the number of JPEGs per second depending on connection (only requested the next frame after the current one arrived or a timeout occurred).

This got me thinking about video calls, which have be notoriously bad on bad connections. Half the time I am just streaming a screen with static information on it, we're not watching videos together. And yet the streaming pipeline is optimised as this article suggests for the higher bandwidth modes - when we're never really using it at all.

The most important part about a video call is rarely the video, is usually the audio. It's counter-intuitive but you are better off having your call without video than you are without sound, and yet when the video falls over it takes the audio with it. Insanity!

Re: We replaced H.264 streaming with JPEG screenshots (and it worked better)

#282

Post by KptMarchewa »

binocarlos wrote: post #11

> I mashed F5 like a degenerate. I love the style of this blog-post, you can really tell that Luke has been deep down in the rabbit hole, encountered the Balrog and lived to tell the tale.

That's amazing! You're going to see a lot more of those AI generated blogs in the coming century!

Re: We replaced H.264 streaming with JPEG screenshots (and it worked better)

#283

Post by kiririn7 »

"By the time you see a bug, the AI has already committed it to main" does anybody actually actively watch the code their agent is writing? i am watching movie recaps on my 2nd monitor. this seems like a problem that they assume exists because they dont actually use their product

Re: We replaced H.264 streaming with JPEG screenshots (and it worked better)

#284

Post by rcarmo »

j45 wrote: post #87
rcarmo wrote: post #7

This was the most entertaining thing I read all day. Kudos. I've had similar experiences in the past when trying to do remote desktop streaming for digital signage (which is not particularly demanding in bandwidth terms). Multicast streaming video was the most efficent, but annoying to decode when you dropped data. I now wonder how far I could have gone with JPEGs...

If playing with Chromecast types multicast or streaming one frame at a time manually worked pretty good.

I have already started hacking at a proof of concept… let’s see how fun it turns out to be.

Re: We replaced H.264 streaming with JPEG screenshots (and it worked better)

#285

Post by kiririn7 »

toledocavani wrote: post #192

This thread is great, truly the only way to get great answers on the HN is to post a wrong blog. But stupid wrong blogs are unlikely to get into HN front page, kudos for the writer for striking the right balance between easy to understand, working, interesting but faulty solution.

so true

Re: We replaced H.264 streaming with JPEG screenshots (and it worked better)

#286

Post by SunlitCat »

lewq wrote: post #221

Hi, author of the post here. Just fixed up some formatting issues from when we copied it into substack, sorry about that. Yeah, I used Opus 4.5 to help me write it (and it actually made me laugh!). But the struggle was real. Something I didn't make clear enough in the post is that jpeg works because each screenshot is taken exactly when it's requested. Whereas streaming video is pushing a certain frame rate. The clie…

Hi lewq, commentator of your post here.

Yeah, I used ChatGPT to help me write this answer ;) (Unlike JPEGs, it works at the right abstraction level for text.)

I think the core issue isn’t push vs pull or frame scheduling, but why you’re sending frames at all. Your use case reads much more like replicating textual/stateful UI than streaming video.

The fact that JPEG “works” because the client pulls frames on demand is kind of the tell — you’ve built a demand-driven protocol, then used it to fetch pixels. That avoids queuing, sure, but it’s also sidestepping video semantics you don’t actually need.

Most of what users care about here is text, cursor position, scroll state, and low interaction latency. JPEG succeeds not because it’s old and robust, but because it accidentally approximates an event-driven model.

Totally fair points about UDP + Kubernetes + enterprise ingress. But those same constraints apply just as well to structured state updates or terminal-style protocols over HTTPS — without dragging a framebuffer along.

Pragmatic solution, real struggle — but it feels like a text/state problem being forced through a video abstraction, and JPEG is just the least bad escape hatch.

— a human (mostly)

Re: We replaced H.264 streaming with JPEG screenshots (and it worked better)

#287

Post by krater23 »

adamjs wrote: post #63

They might want to check out what VNC has been doing since 1998– keep the client-pull model, break the framebuffer up into tiles and, when client requests an update, perform a diff against last frame sent, composite the updated tiles client-side. (This is what VNC falls back to when it doesn’t have damage-tracking from the OS compositor) This would really cut down on the bandwidth of static coding terminals where 90%…

Maybe it would be easier to just USE VNC instead. But the mentioned they have written their software in Rust. Looks like nothig is good enough for Rust coders, they need to fail by reprogramming their things in Rust before they accept that there are still tools for exactly that.

Re: We replaced H.264 streaming with JPEG screenshots (and it worked better)

#288

Post by mdavid626 »

Dylan16807 wrote: post #227
mdavid626 wrote: post #100

Earlier quoted context omitted.

Setting to 1 FPS might not be enough. GOP or P frame setting needs to be adjusted to make every frame keyframe.

Why would you do that? Nearly-static content is where you want even fewer keyframes than usual. In a situation like this you need them when the connection is interrupted and you reset things, and not much of anywhere else.

1 FPS with GOP 60 might just simply not play in some players.

Re: We replaced H.264 streaming with JPEG screenshots (and it worked better)

#289

Post by krater23 »

"By the time you see a bug, the AI has already committed it to main"

Beside of that the Author has no plan at all about encoding, mjpeg, vnc,....

Really, THIS is the product that they sell?! This sounds like a horrible work. Observing a coding agent that does my job, but faster and crappier than me and stopping it when it does totally bullshit to prevent it from commiting to main?

Re: We replaced H.264 streaming with JPEG screenshots (and it worked better)

#290

Post by adastra22 »

martinald wrote: post #204
ryukoposting wrote: post #142

Earlier quoted context omitted.

Of all the suggestions in the comments here, this seems like the best one to start with. Also... I get that the dumb solution to "ugly text at low bitrates" is "make the bitrate higher." But still, nobody looked at a 40M minimum and wondered if they might be looking at this problem from the wrong angle entirely?

In fairness VNC-style approaches are bloody awful even over my 2.5gbit/sec lan on very fast hardware. It just cannot do 4K well (not sure if they need 4k or not). I spent some time compiling the "new" xrdp with x264 and it is incredibly good, basically cannot really tell that I'm remote desktoping. The bandwidth was extremely low as well. You are correct on that part, 40mbit/sec is nuts for high quality. I suspect if…

They are streaming AI coding agents. They are not streaming 4K video.
Post reply on HN