TuneNN: A transformer-based network model for pitch detection

#1

Post by CMLab » Tue, Dec 19, 2023, 12:27 PM UTC

TuneNN: A transformer-based network model for pitch detection
github.com

Re: TuneNN: A transformer-based network model for pitch detection

#2

Post by CMLab » Tue, Dec 19, 2023, 12:27 PM UTC

A transformer-based network model, pitch tracking for musical instruments.

The timbre of musical notes is the result of various combinations and transformations of harmonic relationships, harmonic strengths and weaknesses, instrument resonant peaks, and structural resonant peaks over time.

It utilizes the transformer-based tuneNN network model for abstract timbre modeling, supporting tuning for 12+ instrument types.

Re: TuneNN: A transformer-based network model for pitch detection

#3

Post by vessenes » Tue, Dec 19, 2023, 4:11 PM UTC

This is cool! The very best software-based tuning tech out there is probably in piano tuning apps; they cost hundreds of dollars+ and are specifically made to report on harmonics and other piano nuances.

Do you have any comparisons against other pitch detection tech? Accuracy? Delay/Responsiveness? I assume it's much more compute work than a handcoded FFT type pitch detector.

I think it's possible this would find utilization in the piano world if the output offers something new / something that can analyze what a piano tuning maestro can hear and make it accessible to a mid-tier tuner.

Re: TuneNN: A transformer-based network model for pitch detection

#4

Post by jansommer » Tue, Dec 19, 2023, 4:37 PM UTC

vessenes wrote: post #3

This is cool! The very best software-based tuning tech out there is probably in piano tuning apps; they cost hundreds of dollars+ and are specifically made to report on harmonics and other piano nuances. Do you have any comparisons against other pitch detection tech? Accuracy? Delay/Responsiveness? I assume it's much more compute work than a handcoded FFT type pitch detector. I think it's possible this would find uti…

Sounds like you know a thing or two about pitch detection... I've been working on a C implementation of YIN and PYIN (a real GPL minefield for someone wanting to provide the end result as MIT/public domain!), and am wondering if it's a good choice for real time, cpu-bound speech pitch detection, or if there's better ways. May I ask what your thoughts are on this?

Re: TuneNN: A transformer-based network model for pitch detection

#5

Post by ronsor » Tue, Dec 19, 2023, 4:42 PM UTC

jansommer wrote: post #4

vessenes wrote: post #3

This is cool! The very best software-based tuning tech out there is probably in piano tuning apps; they cost hundreds of dollars+ and are specifically made to report on harmonics and other piano nuances. Do you have any comparisons against other pitch detection tech? Accuracy? Delay/Responsiveness? I assume it's much more compute work than a handcoded FFT type pitch detector. I think it's possible this would find uti…

Sounds like you know a thing or two about pitch detection... I've been working on a C implementation of YIN and PYIN (a real GPL minefield for someone wanting to provide the end result as MIT/public domain!), and am wondering if it's a good choice for real time, cpu-bound speech pitch detection, or if there's better ways. May I ask what your thoughts are on this?

Have you also considered implementing the Nebula[1] algorithm?

[1] https://github.com/Sleepwalking/nebula

Re: TuneNN: A transformer-based network model for pitch detection

#6

Post by azinman2 » Tue, Dec 19, 2023, 4:47 PM UTC

CMLab wrote: post #2

A transformer-based network model, pitch tracking for musical instruments. The timbre of musical notes is the result of various combinations and transformations of harmonic relationships, harmonic strengths and weaknesses, instrument resonant peaks, and structural resonant peaks over time. It utilizes the transformer-based tuneNN network model for abstract timbre modeling, supporting tuning for 12+ instrument types.

This smells like an automated summary.

Re: TuneNN: A transformer-based network model for pitch detection

#7

Post by ks2048 » Tue, Dec 19, 2023, 4:49 PM UTC

vessenes wrote: post #3

This is cool! The very best software-based tuning tech out there is probably in piano tuning apps; they cost hundreds of dollars+ and are specifically made to report on harmonics and other piano nuances. Do you have any comparisons against other pitch detection tech? Accuracy? Delay/Responsiveness? I assume it's much more compute work than a handcoded FFT type pitch detector. I think it's possible this would find uti…

That's interesting. Can you point to one of these piano tuning apps that are $100+?

Re: TuneNN: A transformer-based network model for pitch detection

#8

Post by rrherr » Tue, Dec 19, 2023, 4:53 PM UTC

How does the accuracy of this compare to CREPE?

https://github.com/marl/crepe

https://github.com/maxrmorrison/torchcrepe

Does anyone know what the current state of the art is, within the Music Information Retrieval community?

Re: TuneNN: A transformer-based network model for pitch detection

#9

Post by jansommer » Tue, Dec 19, 2023, 4:53 PM UTC

ronsor wrote: post #5

jansommer wrote: post #4

Earlier quoted context omitted.

Sounds like you know a thing or two about pitch detection... I've been working on a C implementation of YIN and PYIN (a real GPL minefield for someone wanting to provide the end result as MIT/public domain!), and am wondering if it's a good choice for real time, cpu-bound speech pitch detection, or if there's better ways. May I ask what your thoughts are on this?

Have you also considered implementing the Nebula[1] algorithm? [1] https://github.com/Sleepwalking/nebula

I need non-GPL libraries as a reference. The problem with YIN and especially PYIN is that the MIT-code I've found sometimes looks a bit too similar to earlier code in GPL. Rewriting that into the same but in different code is fairly hard. Here I'm assuming that translating eg. GPL Python or C++ into C would mean the license is retained

Re: TuneNN: A transformer-based network model for pitch detection

#10

Post by ranting-moth » Tue, Dec 19, 2023, 4:54 PM UTC

To the dev: the tuner gives me an incredibly high error window with the following message. It doesn't prompt to access the mic (I think that's related). Ubuntu/KDE/Firefox: