Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Debugging audio artifacts caused by... a serial port?

recall.ai

Post reply on HN
Print view View original

21–30 of 33 posts

Re: Debugging audio artifacts caused by... a serial port?

#21

Post by rcxdude »

anyfoo wrote: post #17
theamk wrote: post #12

Earlier quoted context omitted.

it's remote meeting infrastructure, so the latency is critical. When burning CDs, or playing music, it's OK to have a second or two of buffer. When doing conference call, a second of buffer means a second of latency, which means you ask a question and get a response 2 seconds back, which is pretty bad experience. And that's why conference software tries to keep latency as low as possible. (Now, why does it produce a…

Even if the buffer is large enough, at some point (i.e. a long enough meeting in this case), it will fill up. > (Now, why does it produce a pop as opposed to silence/hiccup/stretched sound? probably because it was easiest to code) Sudden "silence" pretty much is a pop, and so is the silence suddenly ending. The sharp transition at least theoretically contains energy in all frequencies (or rather the full bandwidth of…

It's possible to handle buffer underruns more elegantly than that, but it does require more processing power on the receive side of the buffer (basically by using some strategy to extrapolate the audio forward and decay, as opposed to just dropping the signal to zero when there's no data coming from the other side). It's a common thing to do in streaming audio contexts, especially voice, but generally at the end of the user's network connection which is presumed to be unreliable, not in the middle of a processing pipeline which is presumed to be able to hit its latency targets.

Re: Debugging audio artifacts caused by... a serial port?

#22

Post by anyfoo »

rcxdude wrote: post #21
anyfoo wrote: post #17

Earlier quoted context omitted.

Even if the buffer is large enough, at some point (i.e. a long enough meeting in this case), it will fill up. > (Now, why does it produce a pop as opposed to silence/hiccup/stretched sound? probably because it was easiest to code) Sudden "silence" pretty much is a pop, and so is the silence suddenly ending. The sharp transition at least theoretically contains energy in all frequencies (or rather the full bandwidth of…

It's possible to handle buffer underruns more elegantly than that, but it does require more processing power on the receive side of the buffer (basically by using some strategy to extrapolate the audio forward and decay, as opposed to just dropping the signal to zero when there's no data coming from the other side). It's a common thing to do in streaming audio contexts, especially voice, but generally at the end of t…

Oh yeah, I didn’t mean to imply that the pops are a necessary consequence of buffer overruns. But as you say, gracefully mitigating the symptom requires non-negligible engineering effort (and potentially resources), when the actual problem, namely buffer overruns occurring because the consumer is too slow, shouldn’t exist in this particular system in the first place.

Re: Debugging audio artifacts caused by... a serial port?

#23

Post by rcxdude »

PaulDavisThe1st wrote: post #15

> There are a few processes in the bot that are especially latency sensitive, which we have tuned the nice value for. This immediately signifies some level of "does not understand how this stuff works". Latency sensitive (audio or other) tasks need to be in the SCHED_RR or SCHED_FIFO scheduling class, which nice(1) has no effect on. Conversely, using nice(1) on a SCHED_OTHER task is also unlikely to work, given that…

I think in this case, unless they were running an RT kernel, it wouldn't have helped, since the interrupt hogging the CPU was non-preemptable. But it's good advice in general.

Re: Debugging audio artifacts caused by... a serial port?

#24

Post by theamk »

anyfoo wrote: post #22
rcxdude wrote: post #21

Earlier quoted context omitted.

It's possible to handle buffer underruns more elegantly than that, but it does require more processing power on the receive side of the buffer (basically by using some strategy to extrapolate the audio forward and decay, as opposed to just dropping the signal to zero when there's no data coming from the other side). It's a common thing to do in streaming audio contexts, especially voice, but generally at the end of t…

Oh yeah, I didn’t mean to imply that the pops are a necessary consequence of buffer overruns. But as you say, gracefully mitigating the symptom requires non-negligible engineering effort (and potentially resources), when the actual problem, namely buffer overruns occurring because the consumer is too slow, shouldn’t exist in this particular system in the first place.

The delays was on their servers, not on endpoints. So from the consumers' view, this was a buffer underrun due bad communication link.

(In reality, the communication link was fine and it was packet-forwarding server that was slow; but I bet the consumers just saw the packet delays)

Re: Debugging audio artifacts caused by... a serial port?

#25

Post by saagarjha »

Animats wrote: post #19

Bottlenecking on logging is a common problem. Classically, Linux assumed it has a serial console device. It can still be enabled at kernel compile time.[1] Apparently, this mass of AWS instances, VMs, and Docker images works that way. I liked the QNX approach, where the kernel sends log messages to another process, loaded on boot. When you build a boot image, you provide a logger process to read those messages. There…

How do you debug kernel boot before that process is started?

Re: Debugging audio artifacts caused by... a serial port?

#27

Post by kazinator »

saagarjha wrote: post #25
Animats wrote: post #19

Bottlenecking on logging is a common problem. Classically, Linux assumed it has a serial console device. It can still be enabled at kernel compile time.[1] Apparently, this mass of AWS instances, VMs, and Docker images works that way. I liked the QNX approach, where the kernel sends log messages to another process, loaded on boot. When you build a boot image, you provide a logger process to read those messages. There…

How do you debug kernel boot before that process is started?

In terms of print debugging, one way is by using lower level printing routines, like something specific to the target system you're using that sends bytes directly to a serial port.

Re: Debugging audio artifacts caused by... a serial port?

#29

Post by PaulDavisThe1st »

rcxdude wrote: post #23
PaulDavisThe1st wrote: post #15

> There are a few processes in the bot that are especially latency sensitive, which we have tuned the nice value for. This immediately signifies some level of "does not understand how this stuff works". Latency sensitive (audio or other) tasks need to be in the SCHED_RR or SCHED_FIFO scheduling class, which nice(1) has no effect on. Conversely, using nice(1) on a SCHED_OTHER task is also unlikely to work, given that…

I think in this case, unless they were running an RT kernel, it wouldn't have helped, since the interrupt hogging the CPU was non-preemptable. But it's good advice in general.

RT is in the mainline kernel now. Run with threaded_irqs, all interrupt handlers become preemptable (by default).

Re: Debugging audio artifacts caused by... a serial port?

#30

Post by Animats »

saagarjha wrote: post #25
Animats wrote: post #19

Bottlenecking on logging is a common problem. Classically, Linux assumed it has a serial console device. It can still be enabled at kernel compile time.[1] Apparently, this mass of AWS instances, VMs, and Docker images works that way. I liked the QNX approach, where the kernel sends log messages to another process, loaded on boot. When you build a boot image, you provide a logger process to read those messages. There…

How do you debug kernel boot before that process is started?

In QNX? The boot loader loads the kernel and whatever other processes you put in the boot image. They all start as soon as the kernel starts. The kernel itself is just memory allocation, message passing, CPU dispatching, and timers. It has no I/O at all, no persistent state, and it's passive - any activity initiation has to come from user space. It doesn't even have strings.

Actual kernel debugging is rarely needed except on strange or broken hardware. If you have to do that, you use a JTAG debugger.

Further startup is handled by a startup process in the boot image. It's running in user space, and loads more drivers, file systems, networking, etc. Anything it needs to log it sends to a user space logger process that was also part of the boot image. What that does depends on the target hardware. Often, there's no console or display in embedded. It might send on a network. Or write to a circular buffer that can be read out later.

Post reply on HN