Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Android’s 10 Millisecond Problem explained

superpowered.com

Post reply on HN
Print view View original

131–140 of 311 posts

Re: Android’s 10 Millisecond Problem explained

#131

Post by cnvogel »

ynniv wrote: post #49

TL;DR the Linux layer (ALSA) and the Java layer (Audio Flinger) use widely compatible but high latency techniques, whereas Apple designed their API's and hardware such that these layers can be optimized to almost nothing. (From the article: http://superpowered.com/wp-content/uploads/2015/04/Android-A... )

ALSA is not the problem, it works very fine for the low-latency case, I can reliably run soundcards with light processing load at 96 kHz @ 64 frames/period (20.7ms --> 1.4 ms latency) on a quad core i5, e.g. for running a reverb effect, but most of the time I only record and will settle for 1024 frames/period or so. (210ms --> 20ms). The period size, just for completeness, is the number of samples recorded on each block that is forwarded to the audio processing application.

If whatever audio framework you use doesn't allow to run processing with a input-to-output delay (latency) of two times the period size, it's broken (probably the case for Audio Flinger at Android, don't know much about it).

    ➜  ~  jackd -d alsa -p 64 -r 96000
    jackdmp 1.9.10
    Copyright 2001-2005 Paul Davis and others.
    Copyright 2004-2014 Grame.
    (...)
    creating alsa driver hw:0|hw:0|64|2|96000|0|0|nomon|swmeter|-|32bit
    configuring for 96000Hz, period = 64 frames (0.7 ms), buffer = 2 periods
    ALSA: final selected sample format for capture: 32bit integer little-endian
    ALSA: use 2 periods for capture
    ALSA: final selected sample format for playback: 32bit integer little-endian
    ALSA: use 2 periods for playback
(this is on my laptop, just for illustration purposes)

Re: Android’s 10 Millisecond Problem explained

#132

Post by BFay »

hrnnnnnn wrote: post #87
ohitsdom wrote: post #73

Is 10ms really that big of a deal? I'm an amateur musician so have some experience playing in bands, but I have a hard time believing 10ms would feel off when playing with others.

When recording guitar on my computer, I can feel the difference between 10ms and 5ms. 10 isn't horrible, but it's definitely not optimal.

If you think about it, speed of sound is about 340.29 meters per second... also 0.34029 meters per millisecond... also 1.11644 feet per millisecond.

So 5ms vs 10ms latency is like the difference between having your amp 5 feet or 10 feet away.

Re: Android’s 10 Millisecond Problem explained

#133

Post by on_and_off »

izacus wrote: post #121
Synaesthesia wrote: post #113

Earlier quoted context omitted.

I'm a musician, 10ms latency would be fine, however as they note, most Android apps have 100ms latency, or 200ms round-trip latency. That is definitely not usable.

Huh, those tests show it has 35-50, not 200ms O.o

I am too lazy to try to find the exact numbers and versions but back in the Android 1.x, 2.x days, the latency was in the 200 ms ballpark. So things have improved a lot since then (even though there is still a lot of ground to cover)

Re: Android’s 10 Millisecond Problem explained

#134

Post by esonderegger »

troymc wrote: post #104

There is something I don't understand; maybe someone here can explain: Sound travels at about 340 m/s (in a typical room). That means it travels about 3.4 metres in 10 milliseconds. Therefore another way to get a 10 millisecond problem is to stand 3.4 metres from the orchestra. Most people sit farther than 3.4 metres from the orchestra, yet they don't complain about a lag between when the violin bow moves and the sou…

Our brains perceive what we hear trailing what we see by small amounts to be completely normal. If you show people a video of an orchestra concert with the sound of a violin coming 50ms before seeing the bow move, most would immediately notice something is off.

Musicians performing together, however, is a much harder problem than just listening. Ask anyone who has ever performed in a DCI-style drum corps, they will tell you compensating for hearing someone on the other side of field 200ms or so late is incredibly difficult.

Re: Android’s 10 Millisecond Problem explained

#135

Post by shub »

troymc wrote: post #104

There is something I don't understand; maybe someone here can explain: Sound travels at about 340 m/s (in a typical room). That means it travels about 3.4 metres in 10 milliseconds. Therefore another way to get a 10 millisecond problem is to stand 3.4 metres from the orchestra. Most people sit farther than 3.4 metres from the orchestra, yet they don't complain about a lag between when the violin bow moves and the sou…

IANA neuroscientist but it seems reasonable that the brain will fix up small errors. It has to anyway, because it has its own input latency. Besides, the article is referring to discrepancies between multiple audio tracks, which don't depend on vision at all.

For a real-world example, listen to 2 TVs several meters apart and tuned to the same channel. At least with OTA or cable you can expect them to be playing ~simultaneously but the skew between the received signals is easily perceptible.

Re: Android’s 10 Millisecond Problem explained

#136

Post by jblow »

izacus wrote: post #121
Synaesthesia wrote: post #113

Earlier quoted context omitted.

I'm a musician, 10ms latency would be fine, however as they note, most Android apps have 100ms latency, or 200ms round-trip latency. That is definitely not usable.

Huh, those tests show it has 35-50, not 200ms O.o

If you actually read the words in the article you will see that that test was the absolute best case they could find.

Re: Android’s 10 Millisecond Problem explained

#137

Post by daddykotex »

S_A_P wrote: post #43

Reading about this takes me back to the late 90s. I had just gotten serious about recording music on my computer and purchased an 866mhz dell pc. I was a using windows 98 and ASIO was still mostly a Steinberg only development and I couldn't afford cubase. I got a hand me down version of cakewalk pro audio 8. I remember the latency of that set up was about 35ms and thinking this is pretty good. There was some degree o…

Is it possible that their Audio API is much simpler due to the fact that they control all of the implementations (Hardware interface) of iOS?

Re: Android’s 10 Millisecond Problem explained

#138

Post by bsurmanski »

kazinator wrote: post #97
likeclockwork wrote: post #74

Earlier quoted context omitted.

Yes, assembly is more flexible than my favorite higher level language. Because if you have access to assembly, you can implement the whatever higher level language or semantics you want. Whereas if you only have a higher level language, you have to work with/around the abstractions baked into it. For realtime applications it's better to at least have access to a low-level, less overhead API.

That is true. For example, in assembly we can create a higher level language with an absolutely air-tight, precisely tracing garbage collector that is free of issues like false retention. We cannot do that in C. That's because there are areas of the program state that are "off limits", and the compiler generates "GC ignorant" code.

do you mind elaborating on this? What would be off limits, and what in assembly would allow a garbage collector free of false retention vs C?

Re: Android’s 10 Millisecond Problem explained

#139

Post by jblow »

cnvogel wrote: post #131
ynniv wrote: post #49

TL;DR the Linux layer (ALSA) and the Java layer (Audio Flinger) use widely compatible but high latency techniques, whereas Apple designed their API's and hardware such that these layers can be optimized to almost nothing. (From the article: http://superpowered.com/wp-content/uploads/2015/04/Android-A... )

ALSA is not the problem, it works very fine for the low-latency case, I can reliably run soundcards with light processing load at 96 kHz @ 64 frames/period (2 0.7ms --> 1.4 ms latency) on a quad core i5, e.g. for running a reverb effect, but most of the time I only record and will settle for 1024 frames/period or so. (2 10ms --> 20ms). The period size, just for completeness, is the number of samples recorded on each…

But ... why is there a period size? Isn't that a broken design that can only introduce latency? What is wrong with "however much audio data is ready when the application asks, send it"?

Re: Android’s 10 Millisecond Problem explained

#140

Post by cnvogel »

stinos wrote: post #29

So if I get it correctly the problem is twofold: there is some extra intermediate processing and the buffer size and sampling rate are fixed to 256 samples and 48kHz respectively? And which of these two does Superpowered fix? Or both? And what would be the lowest possible latency on for example the Nexus 9?

The lowest possible input-to-output latency of an audio workstation is always two times the period size (samples/buffer) (plus a few microseconds of internal delays in the ADCs, controllers, ...)

The sound chips operate on integral periods or a fixed number of samples, and when you get a period worth of audio data from your ADC, the DAC will already have started putting out the first samples of the next period. Hence, you prepare your audio samples for the second-to-next period.

Assume you have audio processing code roughly looking like this:

    while (1) {
        poll(); /* some API function waiting for the "next period" */
        read(soundcard, block_of_samples);  /* or let DMA do it */
        process_samples(block_of_samples);
        write(soundcard, block_of_samples); /* or let DMA do it */
    }
Let's try some ASCII art:

      v- audio samples going into your soundcard
       _     _     _     _     _     _     _     _     _     _     _     _
      /1\   /2\   /3\   /4\   /5\   /6\   /7\   /8\   /9\   /0\   /1\   /2\ ...
         \_/   \_/   \_/v  \_/   \_/   \_/   \_/   \_/   \_/   \_/   \_/   \
                        |
      \________________/\________________/\________________/\________________/
       Period 1          Period 2          Period 3          Period 4
                        |
                        |
                        [*] here, Period 1 has been DMA'ed from the soundcard
                            to an mmapped buffer of your audio application
    
                            here processing of your audio takes place
                        
                                       [*] this is the latest point at which processing
                                         | must complete so that there will be data for
                                         | the soundcard to output. DMA will start
                                         | from the buffer to the soundcard DAC.
                                         |
       _     _     _     _     _     _   v _     _     _     _     _     _
      / \   / \   / \   / \   / \   / \   /1\   /2\   /3\   /4\   /5\   /6\ ...
         \_/   \_/   \_/   \_/   \_/   \_/   \_/   \_/   \_/   \_/   \_/   \
    
      \________________/\________________/\________________/\________________/
    
                                           ^- processed audio samples will come out
                                              of your laptop's speakers...
Obviously, your machine has to be fast enough to do the whole real-time audio computation in a little less than the time between two interrupts, that's the period size. And it must reliably be able to do this, because if it misses the time to have a block of samples ready for the DAC. If it misses that goal, a "underrun" will take place, and the audio application will have to resynchronize, possibly causing some clicking, intermittent audio, ...
Post reply on HN