It's C and Python, not C++
All squares are rectangles I guess.
AudioFlux: A C/C++ library for audio and music analysis
31–40 of 58 posts
Re: AudioFlux: A C/C++ library for audio and music analysis
#32It would be nice to have a comparison with any of the many C++ MIR (music information retrieval) libraries in the wild: - https://essentia.upf.edu/ - https://github.com/marsyas/marsyas - https://github.com/ircam-ismm/pipo - https://github.com/flucoma/flucoma-core/tree/main/include/al...
If a person wanted to transcribe sheet music from recorded audio, do you know which library and features would be the best starting point?
Start with source separation using demucs
Re: AudioFlux: A C/C++ library for audio and music analysis
#33What's this C/C++ language?
mean you are using C++ but only the features that dont suck, so probably 10% of the language, rest is plain C
Re: AudioFlux: A C/C++ library for audio and music analysis
#34Earlier quoted context omitted.
Given what I've seen in audio ML research: 1) Tuning hyperparameters of your audio preprocessing is a pain if it's a preprocessed CPU step. You have to redo preprocessing every time you want to tune your audio feature hyperparams 2) It's quite common to use torchaudio spectrograms, etc. purely because they are faster (I can link to a handful of recent high-impact audio ML github repos if you like) 3) If you use nnAud…
Backpropping filter coefficients sounds clever, but can't you just do that on any layer that takes a spectrum as input?
Backpropping filter coefficients is clever, but it hasn't really caught on much. Google also tried with LEAF (https://github.com/google-research/leaf-audio) to have a learnable audio filterbank.
Anyway, in audio ML what is very common is:
a) Futzing with the way you do feature extraction on the input. (Oh, maybe I want CQT for this task or a different scale Mel etc)
b) Doing feature extraction on generated audio output, and constructing loss functions from generated audio features.
So, as I said, I don't exactly see the utility of this library for deep learning.
With that said, it is definitely nice to have really high speed low latency audio algorithms in C++. I just wouldn't market it as "useful for deep learning" because
a) during training, you need more flexibility than non-GPU methods without backprop
b) if you are doing "deep learning" then your inferred model will presumably be quite large, and there will be a million other things you'll need to optimize to get real-time inference or inference on CPUs to work well.
Is just my gut reaction. It seems like a solid project, I just question the one selling point of "useful for deep learning" that's all.
Re: AudioFlux: A C/C++ library for audio and music analysis
#35It would be nice to have a comparison with any of the many C++ MIR (music information retrieval) libraries in the wild: - https://essentia.upf.edu/ - https://github.com/marsyas/marsyas - https://github.com/ircam-ismm/pipo - https://github.com/flucoma/flucoma-core/tree/main/include/al...
If a person wanted to transcribe sheet music from recorded audio, do you know which library and features would be the best starting point?
I have had mixed luck with this model, which is supposedly state-of-the-art: https://github.com/magenta/mt3
What kind of music are you trying to transcribe?
Feel free to email me.
Re: AudioFlux: A C/C++ library for audio and music analysis
#36What's this C/C++ language?
[deleted]
Re: AudioFlux: A C/C++ library for audio and music analysis
#37Earlier quoted context omitted.
Backpropping filter coefficients sounds clever, but can't you just do that on any layer that takes a spectrum as input?
Backpropping filter coefficients is clever, but it hasn't really caught on much. Google also tried with LEAF ( https://github.com/google-research/leaf-audio ) to have a learnable audio filterbank. Anyway, in audio ML what is very common is: a) Futzing with the way you do feature extraction on the input. (Oh, maybe I want CQT for this task or a different scale Mel etc) b) Doing feature extraction on generated audio ou…
Are there resources you would recommend reading regarding ML and audio?
Re: AudioFlux: A C/C++ library for audio and music analysis
#38Earlier quoted context omitted.
Backpropping filter coefficients is clever, but it hasn't really caught on much. Google also tried with LEAF ( https://github.com/google-research/leaf-audio ) to have a learnable audio filterbank. Anyway, in audio ML what is very common is: a) Futzing with the way you do feature extraction on the input. (Oh, maybe I want CQT for this task or a different scale Mel etc) b) Doing feature extraction on generated audio ou…
Are there resources you would recommend reading regarding ML and audio?
This is a really broad topic. I began studying it about 5 years ago.
Can you start by suggesting what you task you want to do? I'll throw out some suggestions, but you can say something different. Also you are welcome to email me (email in HN profile):
* Voice conversion / singing voice conversion
* Transcription of audio to MIDI
* Classification / tagging of audio scene
* Applying some effect / cleanup to audio
* Separating audio into different instruments
etc
The really quick summary of audio ML as a topic is:
* Often people treat it audio ML as vision ML, by using spectrogram representations of audio. Nonetheless, 1D models are sometimes just as good if not better, but they require very specific familiarity with the audio domain.
* Audio distance measures (loss functions) are pretty crappy and not well-correlated with human perception. You can say the same thing about vision distance measures, but a lot more research has gone into vision models so we have better heuristics around vision stuff. With that said, multi-scale log mel spectrogram isn't that terrible.
* Audio has a handful of little gotches around padding, windowing, etc.
* DSP is a black art and DSP knowledge has high ROI versus just being dumb and black boxy about everything.
Re: AudioFlux: A C/C++ library for audio and music analysis
#39It would be nice to have a comparison with any of the many C++ MIR (music information retrieval) libraries in the wild: - https://essentia.upf.edu/ - https://github.com/marsyas/marsyas - https://github.com/ircam-ismm/pipo - https://github.com/flucoma/flucoma-core/tree/main/include/al...
If a person wanted to transcribe sheet music from recorded audio, do you know which library and features would be the best starting point?
For MIDI:
https://github.com/Music-and-Culture-Technology-Lab/omnizart and https://basicpitch.spotify.com/
They work better if you apply some source separation before (e.g, https://github.com/sigsep/open-unmix-pytorch, https://github.com/facebookresearch/demucs, or https://mvsep.com)
Still, I think the best results are from proprietary models (specifically https://www.ableton.com/en/manual/converting-audio-to-midi/ and https://www.celemony.com/en/melodyne/what-is-melodyne)
Re: AudioFlux: A C/C++ library for audio and music analysis
#40It would be nice to have a comparison with any of the many C++ MIR (music information retrieval) libraries in the wild: - https://essentia.upf.edu/ - https://github.com/marsyas/marsyas - https://github.com/ircam-ismm/pipo - https://github.com/flucoma/flucoma-core/tree/main/include/al...
If a person wanted to transcribe sheet music from recorded audio, do you know which library and features would be the best starting point?
Check out librosa as well