Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Researchers’ AI aligns sheet music with MIDI audio

venturebeat.com

Post reply on HN
Print view View original

31–40 of 68 posts

Re: Researchers’ AI aligns sheet music with MIDI audio

#31

Post by PaulDavisThe1st »

klodolph wrote: post #14
neolog wrote: post #6

Earlier quoted context omitted.

What information isn't captured in the MIDI but is captured in the electronic sheet music?

Tons. Nearly everything. Sheet music is very expressive. So is MIDI. It’s kind of like asking, “What information isn’t captured in the movie but is captured in the novel?” They’re just radically different ways of expressing music, and much is lost when you translate from one to the other. If you are curious, try flipping through a book on orchestration at the library sometime, or a book on music theory. Even at its m…

Depends a lot on the instrument.

For a keyboard instrument, played conventionally, MIDI does a fairly good job of capturing everything that a composer could tell a performer (and vice-versa).

Slightly less true for percussion, but still somewhat true.

Rather untrue for any instrument where technique can be (ab)used to alter timbre significantly (e.g. most reeds, most strings)

Very untrue for any instrument capable of continuous pitch generation (e.g. unfretted strings).

Obviously "arbitrary instructions" are out, but then they are also not formally part of "sheet music", but more "additional written material from the composer", which can accompany MIDI in various ways too.

Re: Researchers’ AI aligns sheet music with MIDI audio

#32

Post by kevingadd »

creato wrote: post #26
klodolph wrote: post #17

Earlier quoted context omitted.

This kind of problem has been extensively studied for years and it turns out that it is not that easy. The fundamental is not the strongest harmonic, the harmonics don't line up, what happens when you play multiple notes at the same time, etc. It is a big stinking mess.

There's a game called Rocksmith that has been on the market for many years that basically does this in real time on live guitar playing with far more than enough accuracy to make the game fun. And that is doing it basically per-note! This task (aligning sheet music with MIDI) would allow for global optimization approaches which should be far easier and more likely to work.

Electric guitar going through a clean (ish) input is a pretty narrowly scoped problem that won't necessarily generalize

Re: Researchers’ AI aligns sheet music with MIDI audio

#33

Post by kevingadd »

PaulDavisThe1st wrote: post #30
modeless wrote: post #12

Earlier quoted context omitted.

It's more the other way around. The MIDI has extra information that obfuscates the original sheet music. Most commonly, timing variation which can be intentional and quite large, but also other kinds of variation and even mistakes. Additionally, the system described in the article works on scanned paper sheet music, not electronic sheet music, so you also have to factor in OCR issues. All things considered it would b…

technically, that would be a case of the MIDI data having noise in it, not extra information.

It seems unreasonable to call something "noise" if it's meant to be kept. Most traditional definitions of that word would not apply to things like syncopation and variation

Re: Researchers’ AI aligns sheet music with MIDI audio

#34

Post by klodolph »

dragontamer wrote: post #28
klodolph wrote: post #17

Earlier quoted context omitted.

This kind of problem has been extensively studied for years and it turns out that it is not that easy. The fundamental is not the strongest harmonic, the harmonics don't line up, what happens when you play multiple notes at the same time, etc. It is a big stinking mess.

I'm no expert, but isn't that just autocorrelation? ( https://en.wikipedia.org/wiki/Autocorrelation ) Autocorrelation is the algorithm used for Radar (at least, that's what my professors told me). When radar bounces off of another object, there are different "echos" of that radar (ex: an object was 10 miles away and 20 miles away: the 20-mile object will take 2x longer to come back). Its messy and everything. The rad…

Autocorrelation is a good start, IF there is only one note playing at a time, and then it SOMETIMES works. You also get multiple peaks in the autocorrelation and it is not obvious which one to choose as the fundamental frequency.

Re: Researchers’ AI aligns sheet music with MIDI audio

#35

Post by pishpash »

amelius wrote: post #20
neolog wrote: post #2

If you already have a MIDI why would you need the sheet music?

Perhaps they used MIDI only for reference during development. In an actual application they wouldn't need the MIDI. Just my guess.

If anyone bothered to click the link, it's not MIDI but MIDI-generated audio (title should be changed). They used MIDI audio because likely it simplified the problem along a number of dimensions: rhythm and timbre are fixed, and there is no noise. It also means it's cheating and it will not work on acoustic performances, but it's a good start. I mean if you can't solve the problem for MIDI audio you will never solve it for anything else.

Re: Researchers’ AI aligns sheet music with MIDI audio

#36

Post by pishpash »

dragontamer wrote: post #28
klodolph wrote: post #17

Earlier quoted context omitted.

This kind of problem has been extensively studied for years and it turns out that it is not that easy. The fundamental is not the strongest harmonic, the harmonics don't line up, what happens when you play multiple notes at the same time, etc. It is a big stinking mess.

I'm no expert, but isn't that just autocorrelation? ( https://en.wikipedia.org/wiki/Autocorrelation ) Autocorrelation is the algorithm used for Radar (at least, that's what my professors told me). When radar bounces off of another object, there are different "echos" of that radar (ex: an object was 10 miles away and 20 miles away: the 20-mile object will take 2x longer to come back). Its messy and everything. The rad…

There is an equivalent problem in audio, which is source localization. That's much simpler than note cognition.

Re: Researchers’ AI aligns sheet music with MIDI audio

#37

Post by pishpash »

iainctduncan wrote: post #24
klodolph wrote: post #17

Earlier quoted context omitted.

This kind of problem has been extensively studied for years and it turns out that it is not that easy. The fundamental is not the strongest harmonic, the harmonics don't line up, what happens when you play multiple notes at the same time, etc. It is a big stinking mess.

And that's before even separating out by instrument! It's really quite incredible what goes on between the ear and the brain for this kind of thing.

Correlative methods are the wrong model by themselves but could be one part of the right model. Cognition happens by correlation, but also by denoising, by adaptive filtering, by interference cancellation, even by predictive generation. Listening is not a passive process.

Re: Researchers’ AI aligns sheet music with MIDI audio

#38

Post by neolog »

PaulDavisThe1st wrote: post #31
klodolph wrote: post #14

Earlier quoted context omitted.

Tons. Nearly everything. Sheet music is very expressive. So is MIDI. It’s kind of like asking, “What information isn’t captured in the movie but is captured in the novel?” They’re just radically different ways of expressing music, and much is lost when you translate from one to the other. If you are curious, try flipping through a book on orchestration at the library sometime, or a book on music theory. Even at its m…

Depends a lot on the instrument. For a keyboard instrument, played conventionally, MIDI does a fairly good job of capturing everything that a composer could tell a performer (and vice-versa). Slightly less true for percussion, but still somewhat true. Rather untrue for any instrument where technique can be (ab)used to alter timbre significantly (e.g. most reeds, most strings) Very untrue for any instrument capable of…

> Very untrue for any instrument capable of continuous pitch generation (e.g. unfretted strings).

What are the dimensions of a note that can be varied on a violin other than pitch and loudness?

Re: Researchers’ AI aligns sheet music with MIDI audio

#39

Post by PaulDavisThe1st »

neolog wrote: post #38
PaulDavisThe1st wrote: post #31

Earlier quoted context omitted.

Depends a lot on the instrument. For a keyboard instrument, played conventionally, MIDI does a fairly good job of capturing everything that a composer could tell a performer (and vice-versa). Slightly less true for percussion, but still somewhat true. Rather untrue for any instrument where technique can be (ab)used to alter timbre significantly (e.g. most reeds, most strings) Very untrue for any instrument capable of…

> Very untrue for any instrument capable of continuous pitch generation (e.g. unfretted strings). What are the dimensions of a note that can be varied on a violin other than pitch and loudness?

Not relevant to the line quoted above, but ... you can apply more or less bow pressure, resulting in different timbres. You can pick/pluck the strings rather than bowing them at all.

Re: Researchers’ AI aligns sheet music with MIDI audio

#40

Post by PaulDavisThe1st »

kevingadd wrote: post #33
PaulDavisThe1st wrote: post #30

Earlier quoted context omitted.

technically, that would be a case of the MIDI data having noise in it, not extra information.

It seems unreasonable to call something "noise" if it's meant to be kept. Most traditional definitions of that word would not apply to things like syncopation and variation

"Timing variations" as mentioned in the comment I was replying to was, in context, "noise". If the performer was adding syncopation or other "planned" variation, then sure, it's information, but I don't think that is what was meant.
Post reply on HN