Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

3K, 60fps, 130ms: achieving it with Rust

blog.tonari.no

Post reply on HN
Print view View original

41–50 of 211 posts

Re: 3K, 60fps, 130ms: achieving it with Rust

#41

Post by clpwn »

jchw wrote: post #3

Nitpick: “audiophile-quality sound” it seems, is becoming the new “military-grade encryption.” I don’t have many other comments to make other than I am surprised rust-analyzer was only mentioned in passing.

You're right, that sounds way too fluffy.

To clarify, we're targeting "transparent" sounding audio, not "FLACs or bust" audio. Right now we send stereo 48kHz 96kb/s Opus (CELT, not SILK) that we found hit the voice transparency sweet-spot compared to the lossless audio source. We had used higher bitrates in the past, and could easily go back to them, but quality plateaued at around 96k in our experimentation.

More than choosing sane transparent-sounding encoding parameters, the biggest difference in fidelity by far was choosing the correct microphones and speakers for accurate reproduction of voices.

Re: 3K, 60fps, 130ms: achieving it with Rust

#42

Post by [deleted] »

Aengeuad wrote: post #22
ccostes wrote: post #7

Aside from the Rust aspect (which is cool!), I can't believe we've come this far and still don't have low-latency video conferencing. Maybe I'm overly sensitive, but people talking over each other and the lack of conversational flow drives me crazy with things like hangouts.

John Carmack always has an interesting point to make about latency: https://twitter.com/ID_AA_Carmack/status/193480622533120001 >I can send an IP packet to Europe faster than I can send a pixel to the screen. How f’d up is that? and to relate to the other post about landlines: https://twitter.com/ID_AA_Carmack/status/992778768417722368 >I made a long internal post yesterday about audio latency, and it included “Many…

[deleted]

Re: 3K, 60fps, 130ms: achieving it with Rust

#43

Post by beowulfey »

vertex-four wrote: post #26
markdog12 wrote: post #12

Earlier quoted context omitted.

There's a section in the article about it: "In the beginning (or: why we're not WebRTC)"

I'm interested in what they are using if not WebRTC - there's several good options in this space (SRT would be my go-to choice), so it'd be really interesting to see if they rolled their own wire protocol or used something else.

They built it from scratch

Re: 3K, 60fps, 130ms: achieving it with Rust

#44

Post by Daishiman »

fsociety wrote: post #30
Joeboy wrote: post #15

Earlier quoted context omitted.

> Audiophile grade at least has roots in high fidelity. Does it though? Audiophiles generally seem to eschew fidelity in favour of something that sounds subjectively nice , including the psychoacoustic effects of spending a lot of money. Eg. they seem very fond of "warmth". If you asked me to make something sound "warm", I'd be applying some soft clipping and dampening the top end, not eliminating sources of distorti…

Our ears are incredibly sensitive sensors and I think attributing warmth to soft clipping and dampening the top end is not a complete picture. Also warmth is just a single quality. I have a pair of very accurate “cold” headphones that I prefer for music and a pair of “warm” headphones for electronic music and gaming. Past the headphones, it is not so much warmth as it is space in the sound for me. My headphone amplif…

But those characteristics are based on objective facts of sound reproduction that can be quantified.

The characteristic of warmth is related to amplification of certain harmonics as well as equalization in the signal. This is fairly well understood by now.

Re: 3K, 60fps, 130ms: achieving it with Rust

#45

Post by masklinn »

jchw wrote: post #3

Nitpick: “audiophile-quality sound” it seems, is becoming the new “military-grade encryption.” I don’t have many other comments to make other than I am surprised rust-analyzer was only mentioned in passing.

> Nitpick: “audiophile-quality sound” it seems, is becoming the new “military-grade encryption.”

It's too bad they didn't explain it. I expected they meant allowance for "full bandwidth" audio (possibly including music you can listen to).

Video conferencing systems generally use voice-only codecs compressed to shit, full of artifacts in the voice range and utterly dead outside of it.

Re: 3K, 60fps, 130ms: achieving it with Rust

#46

Post by qppo »

GuiA wrote: post #37
josh2600 wrote: post #32

Earlier quoted context omitted.

The media lab has done a ton of research on this. I seem to remember people being able to notice visual latency at 30ms and audio latency at 80-120ms (this is because light is faster than sound).

Interesting, would love to read more if specific papers/authors come to your mind. I suspect there's a big gap between e.g. "noticing the audio latency when audio is played as a result of pressing a button" vs "audio latency affecting the flow of a multiparty conversation".

it's probably the latter, because the former is about 5ms (which is equivalent to the statement, "how short of a time between sounds are they perceivable as separate" aka the lower frequency threshold of hearing). It's non obvious that they're the same limit.

Re: 3K, 60fps, 130ms: achieving it with Rust

#47

Post by vlovich123 »

bob1029 wrote: post #39
ccostes wrote: post #7

Aside from the Rust aspect (which is cool!), I can't believe we've come this far and still don't have low-latency video conferencing. Maybe I'm overly sensitive, but people talking over each other and the lack of conversational flow drives me crazy with things like hangouts.

The biggest problem is that of the video codecs which ultimately boils down to using interframe compression. This technique requires that a certain # of video frames be received and buffered before a final image can be produced. This requirement imposes a baseline amount of latency that can never be overcome by any means. It is a hard trade-off in information theory. Something to consider is that there are alternativ…

You can also just configure your video encoder to not use B-frames. Then if you make all consecutive frames P frames then the size is very maintainable. It gets trickier if your transport is lossy since a dropped P frame is a problem but it's not an unsolvable problem if you use LTR frames intelligently.

All the benefits of efficient codecs, more manageable handling of the latency downsides.

The challenges you'll run into instantly with JPEG is that the file size increase & encoding/decoding time on large resolutions outstrips any benefits you get in your limited tests. For video game applications you have to figure out how you're going to pipeline your streaming more efficiently than transferring a small 10 kb image as otherwise you're transferring each full uncompressed frame to the CPU which is expensive. Doing JPEG compression on the GPU is probably tricky. Finally decode is the other side of the problem. HW video decoders are embarrassingly fast & super common. Your JPEG decode is going to be significantly slower.

* EDIT: For your weekend project are you testing it with cloud servers or locally? I would be surprised if under equivalent network conditions you're outperforming Stadia so careful that you're not benchmarking local network performance against Stadia's production on public networks perf.

Re: 3K, 60fps, 130ms: achieving it with Rust

#48

Post by wenc »

onion2k wrote: post #13
ttul wrote: post #4

If this actually works, I am desperately keen to get my hands on it. If you have the capacity for high bandwidth, why not use it? Zoom’s model must work on whatever crappy broadband people have in their home office. If you have gigabit, it doesn’t seem to make use of that extra capacity to improve video quality. As for sound, I don’t think audiophile quality is necessary...

As for sound, I don’t think audiophile quality is necessary... Given you'll need about 10Mbps upstream for 60fps 3K video it seems a little unreasonable not to add on a 320Kpbs (or more) audio stream. It could make this useful for things like streaming music concerts.

Semi-related note: there's work being done at Stanford to make it possible for remote musicians to play together in an ensemble at low latencies.

JackTrip is the resulting software -- not end-user friendly, but apparently it works.

https://ccrma.stanford.edu/groups/soundwire/software/jacktri...

(Some basic numbers: sounds takes 1 ms to travel a foot, every ms is a foot of separation between musicians, 30ms of latency = 30 ft separation = the max for jamming. So 130ms is not low enough.)

Re: 3K, 60fps, 130ms: achieving it with Rust

#49

Post by cesarb »

simias wrote: post #9
jchw wrote: post #3

Nitpick: “audiophile-quality sound” it seems, is becoming the new “military-grade encryption.” I don’t have many other comments to make other than I am surprised rust-analyzer was only mentioned in passing.

As far as I'm concerned "audiophile" has been synonymous with "overpriced placebo" basically forever. Beyond that I wish the article had explain a bit better why it chose these "better-than-std" crates. I'm actually using all the std variants in my projects, I'm curious to know if I'm missing out or if I just happen not to hit on their limitations.

> Beyond that I wish the article had explain a bit better why it chose these "better-than-std" crates.

At least for parking_lot, its README has a long list with its advantages over std: https://github.com/Amanieu/parking_lot/blob/master/README.md

Post reply on HN