Intuitively (to me), it feels like a clever variation of something like Needleman-Wunsch could pull this off without using machine learning (as long as you have the right encodings). Can someone enlighten me?
Researchers’ AI aligns sheet music with MIDI audio
11–20 of 68 posts
Re: Researchers’ AI aligns sheet music with MIDI audio
#12Earlier quoted context omitted.
MIDI is a very poor substitute for sheet music, and vice versa. Think of MIDI as a performance and sheet music as the text being performed. If you want to analyze the music or perform it, you want the sheet music. If you want a computer to perform the music, you want the MIDI.
What information isn't captured in the MIDI but is captured in the electronic sheet music?
It's more the other way around. The MIDI has extra information that obfuscates the original sheet music. Most commonly, timing variation which can be intentional and quite large, but also other kinds of variation and even mistakes.
Additionally, the system described in the article works on scanned paper sheet music, not electronic sheet music, so you also have to factor in OCR issues. All things considered it would be very difficult to manually write an algorithm to do this reliably.
Re: Researchers’ AI aligns sheet music with MIDI audio
#13If you already have a MIDI why would you need the sheet music?
Handing a MIDI printout to a musician who expects sheet music would be like handing a novel to a movie director who expects a script.
Re: Researchers’ AI aligns sheet music with MIDI audio
#14Earlier quoted context omitted.
MIDI is a very poor substitute for sheet music, and vice versa. Think of MIDI as a performance and sheet music as the text being performed. If you want to analyze the music or perform it, you want the sheet music. If you want a computer to perform the music, you want the MIDI.
What information isn't captured in the MIDI but is captured in the electronic sheet music?
Tons. Nearly everything. Sheet music is very expressive. So is MIDI. It’s kind of like asking, “What information isn’t captured in the movie but is captured in the novel?” They’re just radically different ways of expressing music, and much is lost when you translate from one to the other.
If you are curious, try flipping through a book on orchestration at the library sometime, or a book on music theory. Even at its most basic, a score records which notes to play, but MIDI doesn’t even do that—MIDI records the note values only, so you can’t tell the difference between G# and Ab. Then add in all the articulations, dynamics, and arbitrary instructions that a composer can put into a score.
Re: Researchers’ AI aligns sheet music with MIDI audio
#15Does this really need AI? Couldn't you just take the maximum of the fourier transform of the rendering multiplied with the audio? Maybe across multiple dimensions.
Re: Researchers’ AI aligns sheet music with MIDI audio
#16If you already have a MIDI why would you need the sheet music?
The first example I can think of, because I run into it often, is when you use syncopation. It's made even worse if the syncopated line contains chords, and then even more-so if the note lengths within the chord are not robotically static.
Writing that on the MIDI piano roll is slobber proof. But if you take music written that way and view it as sheet music, it is nearly impossible to read because of the many ties that don't even necessarily begin or even end in the same place as the others.
It all ends up looking like layered sheets of snow sliding off your roof. A ribbon of differently-sagging tie-lines making it not only unsightly, but impossible to read.
Why sheet music if you have MIDI already? Because as someone who records, I use MIDI for writing, but I don't expect other musicians to follow my piano rolls when recording guitar, bass, vocals, etc. and I wouldn't want them ever to sound as robotic as MIDI tends to be.
Drums however - that's one area where MIDI piano roll to sheet music actually works flawlessly, even if the notes don't have the usual x| appearance.
Re: Researchers’ AI aligns sheet music with MIDI audio
#17Does this really need AI? Couldn't you just take the maximum of the fourier transform of the rendering multiplied with the audio? Maybe across multiple dimensions.
This kind of problem has been extensively studied for years and it turns out that it is not that easy. The fundamental is not the strongest harmonic, the harmonics don't line up, what happens when you play multiple notes at the same time, etc. It is a big stinking mess.
Re: Researchers’ AI aligns sheet music with MIDI audio
#18Does this really need AI? Couldn't you just take the maximum of the fourier transform of the rendering multiplied with the audio? Maybe across multiple dimensions.
Care to elaborate?
Re: Researchers’ AI aligns sheet music with MIDI audio
#19Intuitively (to me), it feels like a clever variation of something like Needleman-Wunsch could pull this off without using machine learning (as long as you have the right encodings). Can someone enlighten me?
Sheet music is a sequence of images. Maybe you thought sheet music was more like a text file?
Re: Researchers’ AI aligns sheet music with MIDI audio
#20If you already have a MIDI why would you need the sheet music?
Perhaps they used MIDI only for reference during development. In an actual application they wouldn't need the MIDI. Just my guess.