Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Deep Learning for Guitar Effect Emulation

teddykoker.com

Post reply on HN
Print view View original

101–110 of 168 posts

Re: Deep Learning for Guitar Effect Emulation

#101

Post by nitrogen »

TheRealPomax wrote: post #78
alexlarsson wrote: post #30

Earlier quoted context omitted.

I assume by real-time he meant "able to produce samples at a rate equal to or higher than the audio output sample rate".

That's not what real time, means though. Real time processing means taking signals as they come in, and outputting the transformed result such that there is as close to no signal lag as possible. The output can in fact be wildly lower or higher resolution, real-time does not particularly say anything about that. It's all about whether the output plays (for practical purposes) at the perceived "same time" as the input…

I think "rate" in the parent comment was just referring to speed, not sample rate. But yes, latency is critical for anything used during recording or performance. However way back when I used to make my own music I used non-realtime plugins sometimes and it was okay.

Re: Deep Learning for Guitar Effect Emulation

#102

Post by munificent »

I'm not an expert on machine learning or DSP, but I do know just enough of each to suspect this isn't anywhere near as impressive as it seems.

A distortion pedal is essentially just a waveshaper [1]. Think of audio in digital terms as just a series of numbers. A waveshaper is just a simple mathematical function. To apply it, you literally just apply the function to each value in the input stream and there's your output stream. There's no memory or interesting algorithms going on. It's the audio equivalent to calling map() on your list of samples with some lambda to produce a new list of samples.

Of course distortion pedals do that in the analogue domain using circuitry, which has some additional complexity because transistors and diodes and friends don't behave exactly like mathematical functions. There's "sag" and some other physical effects that cause the output to also somewhat depend on previous input.

Even so, that can generally be modelled using a simple convolution. Each output sample is calculated by taking some finite number of previous input samples, multiplying each of them by a weight factor, and then summing the results.

Does that sound like a neural net? It is. That's what we call them convolutional neural networks. Convolution is bread and butter in DSP. You can easily generate one that produces the same effect as some piece of hardware or acoustic environment by running an impulse (a single 1.0 sample surrounded by silence) through the system and then recording the result. That "impulse response" essentially is your set of convolution weights.

So using a deep neural network and then training sounds a lot to me like overkill to me. You could accomplish much the same by using a "depth-1 network" and running an impulse through it.

Caveat, though: I am just a novice here, so there could very well be a lot of subtlety I'm missing out on.

[1]: https://en.wikipedia.org/wiki/Waveshaper

Re: Deep Learning for Guitar Effect Emulation

#103

Post by ratww »

munificent wrote: post #98
ratww wrote: post #10

Earlier quoted context omitted.

This is even more impressive since regular IRs can't duplicate the distortion effect itself, only the frequency response

What is the difference between "distortion itself" and "only the frequency response"? Are you saying the phase response is important?

Impulse responses can only represent linear time-invariant systems. Like delays, reverbs, equalization curves.

Distortion is non-linear, it is something like a max(-1, min(1, input)) function (a waveshaper, like you said), and it produces harmonics when applied to audio signals.

However guitar pedals also have some additional circuitry to "sweeten" the distortion, removing the extra harmonics added by the clipping diodes. Tubescreamers are notable for cutting bass and enhancing mids. An IR is able to capture this. This is important for guitar pedals, and the reason multiple of them exist.

If you capture the impulse response of an overdrive pedal you'll be capturing only the frequency response of a distorted impulse. If you process clean guitar trough this you'll simulate the frequency response but not the distortion itself, so it will just be a clean guitar with a tinny, shrill, sound, not an overdriven guitar sound.

One way around it (other than the idea in this article!) is doing multiple passes of Impulse Response capture with different amplitudes, this will capture this distortion non-linearity. This is supposedly how a Kemper Profiler works.

Re: Deep Learning for Guitar Effect Emulation

#104

Post by cjlars »

samplenoise wrote: post #58
fab1an wrote: post #3

Pretty cool, though I wonder what the latency of this would be if used as a plugin? The author says it works in real-time, but to non music/audio folks this could mean '100 ms latency is real-time enough, right?' Generally, I think the audio VST business is a really fun space to be in for a lifestyle business, as it is way too small to be attractive for VCs. It seems like a space that provides many niches for lots of…

There's latency and there's the somewhat separate question of how much time is needed to make a prediction. Wavenet is causal (no look-ahead) and operates on the sample level so there are no buffers and thus no latency in the strict sense, beyond encoding/decoding into the sample rate and format required by the ML model, which should take <1ms. Whether a model manages to make a prediction in that amount of time depen…

"Round trip," or guitar to processing to speakers needs to be sub 10ms to be transparent to the musician. Source: spent years playing guitar through my guitar -> DAC -> PC -> DAC -> speaker signal chain

Re: Deep Learning for Guitar Effect Emulation

#105

Post by abaga129 »

fab1an wrote: post #85
abaga129 wrote: post #82

Earlier quoted context omitted.

I'm fairly new to the game, but I'm a solo developer. Currently I dont make enough to quit my day job, but it is a nice supplementary income, and it's nice to get paid a bit for something I truly enjoy. There are also several solo/small shop developers that do make a living from selling plug-ins. Here are a few that I can think of off the top of my head. Auburn Sounds: https://www.auburnsounds.com/ Valhalla DSP: http…

what's your link?

Cut Through Recordings: https://cutthroughrecordings.com/home

Re: Deep Learning for Guitar Effect Emulation

#106

Post by samplenoise »

cjlars wrote: post #104
samplenoise wrote: post #58

Earlier quoted context omitted.

There's latency and there's the somewhat separate question of how much time is needed to make a prediction. Wavenet is causal (no look-ahead) and operates on the sample level so there are no buffers and thus no latency in the strict sense, beyond encoding/decoding into the sample rate and format required by the ML model, which should take <1ms. Whether a model manages to make a prediction in that amount of time depen…

"Round trip," or guitar to processing to speakers needs to be sub 10ms to be transparent to the musician. Source: spent years playing guitar through my guitar -> DAC -> PC -> DAC -> speaker signal chain

The receptive field size is how much 'history' the algorithm requires, it doesn't affect the round trip time, which can still be sub ms

Re: Deep Learning for Guitar Effect Emulation

#107

Post by false_kermit »

abaga129 wrote: post #82
whiddershins wrote: post #39

Earlier quoted context omitted.

Do solo or small shop vst plugin developers make any money? I’m curious if anyone has any direct knowledge about that. There are so many professional activities similar to that where no one makes any money and people really just do it for the love, and then there are seemingly similar things like that where people make surprisingly large amounts of money.

I'm fairly new to the game, but I'm a solo developer. Currently I dont make enough to quit my day job, but it is a nice supplementary income, and it's nice to get paid a bit for something I truly enjoy. There are also several solo/small shop developers that do make a living from selling plug-ins. Here are a few that I can think of off the top of my head. Auburn Sounds: https://www.auburnsounds.com/ Valhalla DSP: http…

+1 Valhalla makes some of my favorite reverbs!

Re: Deep Learning for Guitar Effect Emulation

#108

Post by jfkebwjsbx »

TheRealPomax wrote: post #78
alexlarsson wrote: post #30

Earlier quoted context omitted.

I assume by real-time he meant "able to produce samples at a rate equal to or higher than the audio output sample rate".

That's not what real time, means though. Real time processing means taking signals as they come in, and outputting the transformed result such that there is as close to no signal lag as possible. The output can in fact be wildly lower or higher resolution, real-time does not particularly say anything about that. It's all about whether the output plays (for practical purposes) at the perceived "same time" as the input…

RT does not necessarily mean a small latency but a guarantee on a maximum one, whatever that is.

Re: Deep Learning for Guitar Effect Emulation

#109

Post by avisser »

jeffbee wrote: post #100
TheRealPomax wrote: post #78

Earlier quoted context omitted.

That's not what real time, means though. Real time processing means taking signals as they come in, and outputting the transformed result such that there is as close to no signal lag as possible. The output can in fact be wildly lower or higher resolution, real-time does not particularly say anything about that. It's all about whether the output plays (for practical purposes) at the perceived "same time" as the input…

Is that your private definition of "real-time"? I think it is common to define real-time processing by a specified, finite time between input and output. Many real-time processes are concerned more with the consistency of the latency than with its absolute value.

For guitar pedals, there is an implied sub-perceptibility. The output needs to happen as I play - if the delay is too long, it's now a delay pedal.

So realtime might match your definition, but it is consistent in audio production.

For humans, you can start to notice the lag @ 50ms. (A selection of experimental results summarized here https://gamedev.stackexchange.com/a/74975)

Re: Deep Learning for Guitar Effect Emulation

#110

Post by ndm000 »

munificent wrote: post #102

I'm not an expert on machine learning or DSP, but I do know just enough of each to suspect this isn't anywhere near as impressive as it seems. A distortion pedal is essentially just a waveshaper [1]. Think of audio in digital terms as just a series of numbers. A waveshaper is just a simple mathematical function. To apply it, you literally just apply the function to each value in the input stream and there's your outp…

I think the real innovation here is that this was done on just a few minutes of training data, opening up the possibility for all kinds of effects / amps to be modeled through this same method somewhat easily. I'm not sure how current DSPs are designed, but this is likely orders of magnitude more simple than designing the audio transformations (digital or analog) manually.
Post reply on HN