Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

3K, 60fps, 130ms: achieving it with Rust

blog.tonari.no

Post reply on HN
Print view View original

111–120 of 211 posts

Re: 3K, 60fps, 130ms: achieving it with Rust

#111

Post by namibj »

vlovich123 wrote: post #47
bob1029 wrote: post #39

Earlier quoted context omitted.

The biggest problem is that of the video codecs which ultimately boils down to using interframe compression. This technique requires that a certain # of video frames be received and buffered before a final image can be produced. This requirement imposes a baseline amount of latency that can never be overcome by any means. It is a hard trade-off in information theory. Something to consider is that there are alternativ…

You can also just configure your video encoder to not use B-frames. Then if you make all consecutive frames P frames then the size is very maintainable. It gets trickier if your transport is lossy since a dropped P frame is a problem but it's not an unsolvable problem if you use LTR frames intelligently. All the benefits of efficient codecs, more manageable handling of the latency downsides. The challenges you'll run…

Actually, there are commercial CUDA JPEG codecs (both directions) operating at gigapixels per second. It's not a question of speed, but rather the fact that you can at least afford to use H.264's I-frame-only codec for much lower bandwidth requirements.

Re: 3K, 60fps, 130ms: achieving it with Rust

#112

Post by SomeoneFromCA »

masklinn wrote: post #108
SomeoneFromCA wrote: post #68

Earlier quoted context omitted.

What I said actually does make sense. First of all, if you are digitally lowering loudness of audio (say 4 times), you actually are losing precision, and if you later amplify again - you will never return these bits back. This is what is called headroom. So your typical multiply-by-a-floating-point volume control actually kills dynamic range of the sound. I for example never run my OS volume control and players volum…

So what you're saying "actually does make sense" so long as it's a completely different subject than what one would normally assume in context, without you having mentioned such. When people talk about 24 bits (and >48kHz) in the context of "audiophilia", it's generally about the data at rest and "HD audio" (aka 24 bit music files and downloads). Not about the bit depth of the processing pipeline for which it's gener…

Extra bits won't hurt anyway. DAC are not ideal, and there is a comment above about the dynamic range. If your recording is not loud, you lose your dynamic range (say, if your loudest sound is only 40% of the full amplitude, you've already lost more than 1 bit), which can be partially recovered by a higher precision DACs. So it is true in both senses.

Re: 3K, 60fps, 130ms: achieving it with Rust

#113

Post by namibj »

KallDrexx wrote: post #110
Filligree wrote: post #96

Earlier quoted context omitted.

Okay, but I want to wear wireless headphones. Why can't I have both? Wifi doesn't seem to have this latency problem.

I believe RF based wireless headphones (like my Arctis 7 headphones) don't have this latency in them due to not being Bluetooth based. There is some patented codec I think that does allow low latency bluetooth streaming (forgot the name) but that's not heavily implemented in my experience.

Old-school BT headsets are low-latency enough, afaik. But yeah, just blasting the Opus directly from the network to the headphones would solve it, even re-coding in low-latency configuration only adds 5ms.

Re: 3K, 60fps, 130ms: achieving it with Rust

#114

Post by namibj »

sneak wrote: post #94
clpwn wrote: post #41

Earlier quoted context omitted.

You're right, that sounds way too fluffy. To clarify, we're targeting "transparent" sounding audio, not "FLACs or bust" audio. Right now we send stereo 48kHz 96kb/s Opus (CELT, not SILK) that we found hit the voice transparency sweet-spot compared to the lossless audio source. We had used higher bitrates in the past, and could easily go back to them, but quality plateaued at around 96k in our experimentation. More th…

Voice does not extend above 22.05khz, so using sampling rates above 44.1khz is entirely objectively wasteful and useless, unless your codec only works at 48khz input or something. Are you using 48khz for a specific reason?

Please read the official Opus FAQ to sampling rates: https://wiki.xiph.org/OpusFAQ#But_won.27t_the_resampler_hurt...

Re: 3K, 60fps, 130ms: achieving it with Rust

#115

Post by LockAndLol »

After reading it, I'm still not entirely sure what's being done.

Is it live streaming or is it the transport?

Are they doing video encoding (the audio encoding seems to be done by that webrtc-audio thing)?

Have they chosen a progressive encoding format that compresses frames and pumps them out to the wire as soon as they're done?

Is TCP or UDP involved or a new Layer 3 protocol entirely?

Have I just missed all of those parts or were they really missing amid all the Rust celebration?

Re: 3K, 60fps, 130ms: achieving it with Rust

#116

Post by lultimouomo »

vlovich123 wrote: post #47
bob1029 wrote: post #39

Earlier quoted context omitted.

The biggest problem is that of the video codecs which ultimately boils down to using interframe compression. This technique requires that a certain # of video frames be received and buffered before a final image can be produced. This requirement imposes a baseline amount of latency that can never be overcome by any means. It is a hard trade-off in information theory. Something to consider is that there are alternativ…

You can also just configure your video encoder to not use B-frames. Then if you make all consecutive frames P frames then the size is very maintainable. It gets trickier if your transport is lossy since a dropped P frame is a problem but it's not an unsolvable problem if you use LTR frames intelligently. All the benefits of efficient codecs, more manageable handling of the latency downsides. The challenges you'll run…

You can also eliminate I-frames and have I-slices distributed among several P-frames, so that you don't have spikes in bandwidth (and possibly latency if the encoder needs more time to process an I-frames)

Re: 3K, 60fps, 130ms: achieving it with Rust

#117

Post by MR4D »

Gotta love a writeup with this line in it:

  like Brian's 1970s-era MacBook Pro
That's a writer(s) who knows what it's like to read long (aka thorough) technical articles and not bore the readers to death.

Great article!

Re: 3K, 60fps, 130ms: achieving it with Rust

#118

Post by bob1029 »

phoboslab wrote: post #90
bob1029 wrote: post #39

Earlier quoted context omitted.

The biggest problem is that of the video codecs which ultimately boils down to using interframe compression. This technique requires that a certain # of video frames be received and buffered before a final image can be produced. This requirement imposes a baseline amount of latency that can never be overcome by any means. It is a hard trade-off in information theory. Something to consider is that there are alternativ…

Interframes are not a problem, as long as they only reference previous frames, not future ones. I was able to get latency down to 50ms, streaming to a browser using MPEG1[1]. The latency is mostly the result of 1 frame (16ms) delay for a screen capture on the sender + 2-3 frames of latency to get through the OS stack to the screen at the receiving end. En- and decoding was about ~5ms. Plus of course the network laten…

It's funny you mention MPEG1. That's where my journey with all of this began. For MPEG1 testing I was just piping my raw bitmap data to FFMPEG and piping the result to the client browser.

I was never satisfied with the lower latency bound for that approach and felt like I had to keep pushing into latency territory that was lower than my frame time.

That said, MPEG1 was probably the simplest way to get nearly-ideal latency conditions for an interframe approach.

Re: 3K, 60fps, 130ms: achieving it with Rust

#119

Post by ninkendo »

OJFord wrote: post #102
ninkendo wrote: post #89

Earlier quoted context omitted.

Yeah it looks like they wanted to know how long each crate took to build individually. But as long as we're nitpicking, nobody should just pipe `ls` into `xargs` like this, since it fails if anything has spaces in it. Instead, do: for cargo_toml in */Cargo.toml; do crate="$(dirname "${cargo_toml}")" pushd $crate # ... done Don't be that person who writes a script which won't tolerate spaces in filenames!

Alternatively: Don't be that person who clones the repo at a path with spaces in!

Not having spaces in your directory names is certainly a good idea, but I'll be damned if I let any of my code have issues with them. Just because something's a good idea doesn't mean it should be a requirement :)

(The main reason for the advice of "Don't put spaces in paths" is really only because it breaks lots of poorly-written software... but that's not an excuse for your software to be poorly-written!)

Post reply on HN