Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Researchers’ AI aligns sheet music with MIDI audio

venturebeat.com

Post reply on HN
Print view View original

41–50 of 68 posts

Re: Researchers’ AI aligns sheet music with MIDI audio

#41

Post by modeless »

PaulDavisThe1st wrote: post #40
kevingadd wrote: post #33

Earlier quoted context omitted.

It seems unreasonable to call something "noise" if it's meant to be kept. Most traditional definitions of that word would not apply to things like syncopation and variation

"Timing variations" as mentioned in the comment I was replying to was, in context, "noise". If the performer was adding syncopation or other "planned" variation, then sure, it's information, but I don't think that is what was meant.

That is what was meant, in part, as indicated by the comment "which can be intentional". And even unintentional variation can be desired, as it makes the sound less robotic. Noise can be information too.

Re: Researchers’ AI aligns sheet music with MIDI audio

#42

Post by PaulDavisThe1st »

modeless wrote: post #41
PaulDavisThe1st wrote: post #40

Earlier quoted context omitted.

"Timing variations" as mentioned in the comment I was replying to was, in context, "noise". If the performer was adding syncopation or other "planned" variation, then sure, it's information, but I don't think that is what was meant.

That is what was meant, in part, as indicated by the comment "which can be intentional". And even unintentional variation can be desired, as it makes the sound less robotic. Noise can be information too.

My reading of this was different. I felt that the commenter was noting how the timing information in MIDI is not restricted to the note types represented in sheet music, and can include swing and shuffle that could not be represented in that format.

If those timing variations were intentional, then I agree they represent information. If they are just sloppy playing, they are noise. And if the performer is just applying a stylistic form ("swing these bars"), then it's an inefficient encoding of the musical intent.

Re: Researchers’ AI aligns sheet music with MIDI audio

#43

Post by bluGill »

neolog wrote: post #38
PaulDavisThe1st wrote: post #31

Earlier quoted context omitted.

Depends a lot on the instrument. For a keyboard instrument, played conventionally, MIDI does a fairly good job of capturing everything that a composer could tell a performer (and vice-versa). Slightly less true for percussion, but still somewhat true. Rather untrue for any instrument where technique can be (ab)used to alter timbre significantly (e.g. most reeds, most strings) Very untrue for any instrument capable of…

> Very untrue for any instrument capable of continuous pitch generation (e.g. unfretted strings). What are the dimensions of a note that can be varied on a violin other than pitch and loudness?

Not a violin, but on mandolin I can greatly change the tone (which is somehow the overtones) by changing here and how the pick hits the strings. A good violin player can do a lot with the bow.

Re: Researchers’ AI aligns sheet music with MIDI audio

#44

Post by Jasper_ »

the8472 wrote: post #25
iainctduncan wrote: post #23

Earlier quoted context omitted.

This is one of variants of an area called "score following" and it's really hard. I remember about 10-15 years ago there was this craze in start-up land which led to many corpses of start-ups who thought solving semantic music problems would be easy with FFTs. But forgot to talk to an actual MIR (music information retrieval) or Musicologist. I remember a friend of mine working for one about 15 years ago and when she…

That makes sense when you're dealing with raw audio and need to disentangle it. But here we can work in the other direction to using the structure of the sheet to find the right spot in the mess on the other side. I'm sure there are further difficulties, just saying that your answer isn't entirely satisfying.

Sheet music has far less structure than you think it does.

Re: Researchers’ AI aligns sheet music with MIDI audio

#45

Post by klodolph »

neolog wrote: post #38
PaulDavisThe1st wrote: post #31

Earlier quoted context omitted.

Depends a lot on the instrument. For a keyboard instrument, played conventionally, MIDI does a fairly good job of capturing everything that a composer could tell a performer (and vice-versa). Slightly less true for percussion, but still somewhat true. Rather untrue for any instrument where technique can be (ab)used to alter timbre significantly (e.g. most reeds, most strings) Very untrue for any instrument capable of…

> Very untrue for any instrument capable of continuous pitch generation (e.g. unfretted strings). What are the dimensions of a note that can be varied on a violin other than pitch and loudness?

In general, think of this: You are touching a musical instrument. You are physically touching the strings, and you can undoubtedly imagine many different ways of touching the strings that would have different effects on the sound produced. If you were to sit down and imagine different ways to play violin, you would probably come up with techniques that already have standardized names (in Italian, of course) and perhaps even standardized symbols in notation. The possibilities are AMAZING and definitely not limited to simple pitch/loudness (I mean, obviously!)

If you want to see a reference, go to your local library and find a book like The Study of Orchestration (https://www.amazon.com/Study-Orchestration-Fourth-Samuel-Adl...) and flip to the section on strings. I own this book, but I don’t know where it is at the moment, so… off the top of my head, here are other dimensions besides pitch and loudness:

- Notes can be connected in different ways. Legato, détaché, martelé, staccato, spiccato, sautillé, jeté/ricochet, tremolo, pizzacato, louré, marcato

- Bowing direction: up/down (they sound different)

- Adjust bow position: sul ponticello, sul tasto

- Natural and artificial harmonics

- Use of mutes

- “Extended techniques”—altering the tuning, col legno, etc.

These all affect the sound. Note that the difference between staccato and legato is NOT accounted for solely by the length of the notes as in a MIDI file. You also might be surprised how many of these have really boring, everyday notational conventions. As in, a violinist would look at sheet music and say, “obviously, technique X is used for this note”, but that would not be encoded in the MIDI file at all.

All of the above techniques are explained and demonstrated in YouTube videos if you are interested.

Re: Researchers’ AI aligns sheet music with MIDI audio

#46

Post by jcranmer »

neolog wrote: post #38
PaulDavisThe1st wrote: post #31

Earlier quoted context omitted.

Depends a lot on the instrument. For a keyboard instrument, played conventionally, MIDI does a fairly good job of capturing everything that a composer could tell a performer (and vice-versa). Slightly less true for percussion, but still somewhat true. Rather untrue for any instrument where technique can be (ab)used to alter timbre significantly (e.g. most reeds, most strings) Very untrue for any instrument capable of…

> Very untrue for any instrument capable of continuous pitch generation (e.g. unfretted strings). What are the dimensions of a note that can be varied on a violin other than pitch and loudness?

In a piano, the relevant acoustic properties controllable by the player is key down (with differing amounts of force) and key up, with the pedals effectively acting as a way to delay key up (although one pedal causes the hammers to hit two instead of three strings, which largely manifests as quieting the piano). The simple note on/off + note velocity of MIDI models this well, and it's basically the use case MIDI was designed for.

By contrast, the technique involved in playing the violin family is very varied:

* Which string are you playing the note on? You get different sympathetic vibrations depending on which one you choose.

* Are you moving the bow up or down? Direction matters!

* How are you changing bow movement between notes? Keeping the same direction, or changing? Resting your bow on the string the entire time, or bouncing the bow?

* Where you are playing the bow? Down by the bridge, or up by the fingerboard?

* Are you even using the bow to the play notes? You can pluck it instead!

* Fingering too, you can make many small motions with your finger to give it a vibrating quality.

* On a related note, the pitch you play is a continuous quality: you can slide from one note to the other and hit all the notes in between. Unlike a piano (and MIDI), where notes are discrete pitches.

* You can also adjust tuning of the strings (though not on the fly), or damper them with a mute (which can be done during a long rest in a piece).

There's probably a few more expressive techniques I've forgotten, and I've definitely forgotten all of the fancy Italian names for these techniques.

You can look at the violin sections of Saint-Saëns' Danse Macabre (https://www.youtube.com/watch?v=71fZhMXlGT4) to see how different bowing techniques can produce rather stark effects.

Re: Researchers’ AI aligns sheet music with MIDI audio

#47

Post by cordellwren »

PaulDavisThe1st wrote: post #42
modeless wrote: post #41

Earlier quoted context omitted.

That is what was meant, in part, as indicated by the comment "which can be intentional". And even unintentional variation can be desired, as it makes the sound less robotic. Noise can be information too.

My reading of this was different. I felt that the commenter was noting how the timing information in MIDI is not restricted to the note types represented in sheet music, and can include swing and shuffle that could not be represented in that format. If those timing variations were intentional, then I agree they represent information. If they are just sloppy playing, they are noise. And if the performer is just applyi…

Consistent microtiming deviations or other forms of "expressive timing" are an essential element to "groove" or rhythmic "feel" in almost all modern popular styles of music. Things can get even crazier with world music styles like Samba, and let's not even get started with classical rubato.

And yet notating any of that information in every bar as, say, three-quarters to one-and-a-half hemidemisemiquavers early or late to each offbeat defeats the purpose of having a readable score.

The score is a reduction that's intentionally lacking tons of information that defines the essence of a performance. That's not a bug, but a feature.

Re: Researchers’ AI aligns sheet music with MIDI audio

#48

Post by Kye »

jcranmer wrote: post #46
neolog wrote: post #38

Earlier quoted context omitted.

> Very untrue for any instrument capable of continuous pitch generation (e.g. unfretted strings). What are the dimensions of a note that can be varied on a violin other than pitch and loudness?

In a piano, the relevant acoustic properties controllable by the player is key down (with differing amounts of force) and key up, with the pedals effectively acting as a way to delay key up (although one pedal causes the hammers to hit two instead of three strings, which largely manifests as quieting the piano). The simple note on/off + note velocity of MIDI models this well, and it's basically the use case MIDI was…

I'm partial to glissando.

Re: Researchers’ AI aligns sheet music with MIDI audio

#49

Post by PaulDavisThe1st »

cordellwren wrote: post #47
PaulDavisThe1st wrote: post #42

Earlier quoted context omitted.

My reading of this was different. I felt that the commenter was noting how the timing information in MIDI is not restricted to the note types represented in sheet music, and can include swing and shuffle that could not be represented in that format. If those timing variations were intentional, then I agree they represent information. If they are just sloppy playing, they are noise. And if the performer is just applyi…

Consistent microtiming deviations or other forms of "expressive timing" are an essential element to "groove" or rhythmic "feel" in almost all modern popular styles of music. Things can get even crazier with world music styles like Samba, and let's not even get started with classical rubato. And yet notating any of that information in every bar as, say, three-quarters to one-and-a-half hemidemisemiquavers early or lat…

>The score is a reduction that's intentionally lacking tons of information that defines the essence of a performance.

This is precisely part of my point, or would be if you corrected it. The score contains, either implicitly or explicitly, global performance clues that will tell the performer what to do about timing (and if it doesn't that's because it is expected to come from a conductor or other similar source). It's a highly efficient mechanism precisely because it is a global property of the score (or perhaps locally scoped to sections). Much better than providing timing information for every note (as MIDI would do). The MIDI version is an inefficient means of information transfer.

BTW, things do not get even crazier with "world music styles like samba". Samba is an extremely regular groove that is very easy to understand. This is true of most Afro-Cuban derived rhythmic structures - the complexities come from layering a set of very simple patterns. Things do get "even crazier" with rhythmic traditions from Indian, Balinese and some parts of Africa, places where conventional western ways of describing things really don't do a good job at all.

Re: Researchers’ AI aligns sheet music with MIDI audio

#50

Post by dragonwriter »

PaulDavisThe1st wrote: post #49
cordellwren wrote: post #47

Earlier quoted context omitted.

Consistent microtiming deviations or other forms of "expressive timing" are an essential element to "groove" or rhythmic "feel" in almost all modern popular styles of music. Things can get even crazier with world music styles like Samba, and let's not even get started with classical rubato. And yet notating any of that information in every bar as, say, three-quarters to one-and-a-half hemidemisemiquavers early or lat…

>The score is a reduction that's intentionally lacking tons of information that defines the essence of a performance. This is precisely part of my point, or would be if you corrected it. The score contains, either implicitly or explicitly, global performance clues that will tell the performer what to do about timing (and if it doesn't that's because it is expected to come from a conductor or other similar source). It…

> Samba is an extremely regular groove that is very easy to understand. This is true of most Afro-Cuban derived rhythmic structures

Samba isn’t Afro-Cuban derived, whether in its rhythmic structures or otherwise.

Post reply on HN