Live data from Hacker News

Researchers’ AI aligns sheet music with MIDI audio

venturebeat.com

51–60 of 68 posts

Re: Researchers’ AI aligns sheet music with MIDI audio

#51
post #6
post #3

Earlier quoted context omitted.

MIDI is a very poor substitute for sheet music, and vice versa. Think of MIDI as a performance and sheet music as the text being performed. If you want to analyze the music or perform it, you want the sheet music. If you want a computer to perform the music, you want the MIDI.

What information isn't captured in the MIDI but is captured in the electronic sheet music?

Sheet music always has room for human interpretation.

Re: Researchers’ AI aligns sheet music with MIDI audio

#52
post #6
post #3

Earlier quoted context omitted.

MIDI is a very poor substitute for sheet music, and vice versa. Think of MIDI as a performance and sheet music as the text being performed. If you want to analyze the music or perform it, you want the sheet music. If you want a computer to perform the music, you want the MIDI.

What information isn't captured in the MIDI but is captured in the electronic sheet music?

Dynamics. MIDI has volume, but that's not all there is.

Legato vs detache. Articulation.

All sorts of esoteric annotations, e.g. "Fire the cannon!", or "Release the penguins."

Re: Researchers’ AI aligns sheet music with MIDI audio

#53
post #44
post #25

Earlier quoted context omitted.

That makes sense when you're dealing with raw audio and need to disentangle it. But here we can work in the other direction to using the structure of the sheet to find the right spot in the mess on the other side. I'm sure there are further difficulties, just saying that your answer isn't entirely satisfying.

Sheet music has far less structure than you think it does.

Can you explain what you mean by that? Sheet music would seem to be incredibly structured to me. Unless we’re talking about sheet music hand scribbled on paper?

Re: Researchers’ AI aligns sheet music with MIDI audio

#54
post #53
post #44

Earlier quoted context omitted.

Sheet music has far less structure than you think it does.

Can you explain what you mean by that? Sheet music would seem to be incredibly structured to me. Unless we’re talking about sheet music hand scribbled on paper?

Consider the same piece of "sheet music" delivered by different conductors, eg., Beethoven's 5th:

https://www.youtube.com/watch?v=1lHOYvIhLxo&t=24s https://www.youtube.com/watch?v=9aDEq3u5huA

Sheet music is a guide for human beings who understand "musical context" to, with much leeway, create a cohesive sound.

The sound created does not have a deterministic relationship to what's written, or else, why have any live music at all?

And, more importantly, even if it did deterministically produce the same "sound", that sound is layered to us by our prior familiarity with instruments (, room geometry, etc.).

We aren't deconstructing it by "mere frequency", but by meaningfully parsing out & grouping frequencies. There is no naive algorithm to do this.

Re: Researchers’ AI aligns sheet music with MIDI audio

#55

Earlier quoted context omitted.

That is what was meant, in part, as indicated by the comment "which can be intentional". And even unintentional variation can be desired, as it makes the sound less robotic. Noise can be information too.

My reading of this was different. I felt that the commenter was noting how the timing information in MIDI is not restricted to the note types represented in sheet music, and can include swing and shuffle that could not be represented in that format. If those timing variations were intentional, then I agree they represent information. If they are just sloppy playing, they are noise. And if the performer is just applyi…

[deleted]

Re: Researchers’ AI aligns sheet music with MIDI audio

#56

Earlier quoted context omitted.

Sheet music is a sequence of images. Maybe you thought sheet music was more like a text file?

I should have been more clear. This is what I meant be "correct encodings". I would be surprised (but am welcome to being wrong) if a large amount of sheet music hadn't been encoded somehow.

Sheet music isn't mathematical, ie., there isn't a numerical "encoding". It's has a fair amount of natural semantics (eg., "play like a fantasy"),

There is only an embedding or re-presentation in some less informative form. ie., you have to choose some partial measurement system and apply this to the images of scores to produce a numeric representation.

This is, in part, why a midi-file is always going to sound worse than an orchestra operating from sheet music. The musicians understand the intensions of the composer.

Re: Researchers’ AI aligns sheet music with MIDI audio

#57
I'm going through various books by Arnold Schoenberg. He adds lots of tiny examples that I dont have the skill to play nor the ear to "listen" to in my head.

HN: Is there a tool/service/app that allows me to point at a piece of sheet music and it just plays it? I've tried a couple apps but they don't work well.

Re: Researchers’ AI aligns sheet music with MIDI audio

#58
post #6
post #3

Earlier quoted context omitted.

MIDI is a very poor substitute for sheet music, and vice versa. Think of MIDI as a performance and sheet music as the text being performed. If you want to analyze the music or perform it, you want the sheet music. If you want a computer to perform the music, you want the MIDI.

What information isn't captured in the MIDI but is captured in the electronic sheet music?

MIDI was designed around the piano, which is a very low-dimensional instrument. Due to how it works (hammers that hit the string with a given physical velocity), the only things that affect how a note sounds are: when you play it, what key you play, how hard you hit it, and the exact position of the damper pedal (and other pedals if used)%. Pianos have a mechanical system that abstracts your playing and confines it to these dimensions. These are the exact dimensions that MIDI covers. And this is why these days we have extremely accurate piano samples as well as physical modeling synths that sound amazing with MIDI input.

However, on other instruments, you have a much more direct relationship with the sound. With a violin or guitar, you physically touch the strings - directly or with some device. That already means you have something like 6 degrees of freedom in physical space, not counting fretting. Plus if you use a finger directly, your playing won't sound the same as someone else's with different fingers. The whole action of playing a note is a complex process in time, not a single event, so you can't capture it as one set of parameters. Then there's things like tapping on the body of the instrument, not the strings. You'd basically need a model of the contact surface that touches the string, its physical parameters, and its precise motion in time at high resolution, to accurately capture all possible playing techniques that people actually use.

What we've done with sheet music is standardize many of these techniques into instructions that players can perform. It's obviously only a much lower information version, but it's there.

Of course, if your question is "can we encode everything in sheet music with MIDI" the answer is obviously yes, but it's not standardized. A given virtual instrument could hypothetically implement anything you could write a score for (perhaps with an obscene number of samples or a really clever modeling synths) and be controlled by MIDI, but only a tiny subset of these options have vaguely standardized mappings to MIDI. MIDI is a very limited standard with tons of room for expansion, and only the most basic MIDI features interoperate between manufacturers. You can make MIDI music sound amazing with specific setups (and this is effectively how a lot of modern music is produced), but you can't make General MIDI music (the standardized subset) sound amazing.

The other issue is that sheet music is at a higher abstraction level than MIDI. It leaves more to the interpretation of the performer, but it's designed to be easily interpertable. For the portions of this that MIDI does express at a lower level, it's very hard to infer the higher level representation.

% This is excluding prepared pianos and techniques like reaching in and touching the strings, which are rare; of course technically you can do crazy things to a piano, but in practice we don't.

Re: Researchers’ AI aligns sheet music with MIDI audio

#59
post #26

Earlier quoted context omitted.

There's a game called Rocksmith that has been on the market for many years that basically does this in real time on live guitar playing with far more than enough accuracy to make the game fun. And that is doing it basically per-note! This task (aligning sheet music with MIDI) would allow for global optimization approaches which should be far easier and more likely to work.

Electric guitar going through a clean (ish) input is a pretty narrowly scoped problem that won't necessarily generalize

MIDI music is far cleaner than analogue input from a guitar.

Re: Researchers’ AI aligns sheet music with MIDI audio

#60

Earlier quoted context omitted.

Consistent microtiming deviations or other forms of "expressive timing" are an essential element to "groove" or rhythmic "feel" in almost all modern popular styles of music. Things can get even crazier with world music styles like Samba, and let's not even get started with classical rubato. And yet notating any of that information in every bar as, say, three-quarters to one-and-a-half hemidemisemiquavers early or lat…

>The score is a reduction that's intentionally lacking tons of information that defines the essence of a performance. This is precisely part of my point, or would be if you corrected it. The score contains, either implicitly or explicitly, global performance clues that will tell the performer what to do about timing (and if it doesn't that's because it is expected to come from a conductor or other similar source). It…

I was remarking on your comment that timing variation is noise, as you're relegating far too much into that category due to your narrow view on what counts as musical intent. Rhythmic variations can be highly irregular and even seemingly random, yet follow an underlying logic and be stylistically essential to the performance, which means they would merit notation in some form.

And I'm sorry, but regarding samba, you have absolutely no idea what you're talking about. I assume you're referring only to the surdo's backbeat, but the essence of the samba rhythm is the sixteenth-note groove played by the pandeiro, and that sound is pretty much as far from "extremely regular" as you can get while still maintaining a consistent pulse. I found the style relevant to bring up, as it's a commonly given basic example of a groove featuring microrhythmic variation.

Post reply on HN