Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Blade Runner re-encoded using neural networks

vox.com

Post reply on HN
Print view View original

51–60 of 70 posts

Re: Blade Runner re-encoded using neural networks

#51

Post by DannyBee »

Mithaldu wrote: post #28
dogma1138 wrote: post #3

Can anyone explain what does this encoder actually does and how is it different than any other encoder? The article almost seems to make it sound that this encoder somehow reconstructed the film...

This is possibly the next generation of video/audio/image codecs. What he did was create a specialized compression algorithm that works very well to compress the data that is each frame of Blade Runner, and decompress it (lossily, like mp3) back into a video stream. To put this into perspective: Blade Runner is 117 minutes long. At 25 frames per second, that is 175_500 frames. As he says, the input data he used was 2…

"with compressors that can are so good for one specific set of data that compressor + data is smaller than anything the traditional compressors could create."

This is well, hopeful but probably wrong ... because we already know how to make it smaller for this type of data.

The algorithms used for interframe/intraframe prediction are chosen to tradeoff speed vs size. If you built a really large-scale predictor that was able to generate very small representations for changes, you would get what he's built.

(Note that encoders already complexly select from tons of different prediction algorithms for each set of frames, etc)

We can already do this if someone wanted to. We just don't.

Because it's not fast enough (and nothing in that work changes this)

Would it be useful to apply NN's to video encoders to better select among prediction modes, etc. Probably. But that's already being done, and is not this guy's work.

"And even if its not fast enough, it can still be a very efficient compressor, trading size for CPU. "

The problem you have is not just compression time. It's decompression time. The bitstream of H265/etc is meant to be decodable fast.

What this guy is building is not. If you were to make it so, it would probably look closer to a normal video codec bitstream, and take up that much space.

In fact, he hasn't built anything truly new, he's just using existing papers and making an implementation. He also says, in his masters thesis, that is primarily an artistic exploration.

Even with hardware decoding, you can only make stuff so fast.

TL;DR While interesting, there are people working on the things you are talking about, and it's not this guy (at least in this work).

I would not expect magic here. We already create video codec algorithms by trading off cpu cost and size. The trick is trying to get better size without increasing CPU cost significantly. As these resources change (and remember, moore's law is pretty much dead), the video codecs will change, and videos will get smaller, but you aren't likely to see serious breakthroughs. We already could produce very small videos by applying tremendous amounts of CPU power.

Re: Blade Runner re-encoded using neural networks

#52

Post by squeaky-clean »

ionwake wrote: post #43
rangibaby wrote: post #26

Earlier quoted context omitted.

Re: publishing rights, you can make a cover version of a song using a "compulsory mechanical license", whether the owner likes it or not.

So are remixes allowed, or do you need permission from the original publisher?

For a remix, you would also need the permission to use the specific recording, (which can be declined) as well as rights from the song owner.

For example, If you wanted to cover Viva la Vida by Coldplay, you do it and pay the mechanical licensing fees. If you wanted to sample/remix it the recording of Viva la Vida, you would need to contact Capitol Records, who could decline or request specific payment. If you wanted to sample or remix the performance of Viva la Vida from Super Bowl 50, you would probably need the rights from CBS.

https://loudr.fm/faq

http://www.ascap.com/playback/2011/01/features/limelight.asp...

Re: Blade Runner re-encoded using neural networks

#53

Post by AJ007 »

amelius wrote: post #31
robert_tweed wrote: post #27

Earlier quoted context omitted.

I'm pretty sure the law as it stands is robust enough to deal with these issues on the basis of similarity vs substantive difference. A cover usually sounds quite different to an original, enough so that a fan would certainly not consider them the same thing. OTOH, simply using some well recognised phrases (not necessarily a sample) can infringe on the mechanical rights, which are the copyright on the composition as…

So then the algorithm for creating a cover should include a way to detect whether the song is still within the sphere of similarity. This could be used, for example, to iteratively come to a point where the song is no longer considered similar.

The AI can get as close to the similarity line as possible without infringing on copyright and make the final output even better. This could be applicable for everything from music to patented product designs.

Lyrics excluded, there are plenty of human cases where the cover of a song ends up better and more popular than the original. Hell, many of the most well known singers aren't even very good but just a lot better at marketing than say a classically trained opera singer.

Rather than having a conversational AI that can answer complex question the most immediate and low hanging commercial fruit for AI based companies may very well be along the lines of sucking up intellectual property and spitting out stuff that is as good or better. This might sound like theft but it is exactly what every human content producer is doing.

Re: Blade Runner re-encoded using neural networks

#54

Post by k__ »

Mithaldu wrote: post #28
dogma1138 wrote: post #3

Can anyone explain what does this encoder actually does and how is it different than any other encoder? The article almost seems to make it sound that this encoder somehow reconstructed the film...

This is possibly the next generation of video/audio/image codecs. What he did was create a specialized compression algorithm that works very well to compress the data that is each frame of Blade Runner, and decompress it (lossily, like mp3) back into a video stream. To put this into perspective: Blade Runner is 117 minutes long. At 25 frames per second, that is 175_500 frames. As he says, the input data he used was 2…

Is it like a virtual, self modifying, FPGA for compression?

Re: Blade Runner re-encoded using neural networks

#55

Post by svantana »

buro9 wrote: post #25
svantana wrote: post #23

This article alludes to, but never actually brings up, something much more interesting than the project in question: How can copyright coexist with human-level automatic analysis and synthesis of works of art? For example, Spotify is flooded with "covers" of hit songs that are made to sound as similar to the original as possible. In my understanding, it doesn't matter how similar it sounds, as long as it's "remade",…

Music has that sorted already. There is copyright. There is publishing right. If you make a cover of a recording, it is the latter that you are involved with. You'd have to go to the publishers of a piece of music to license the right to publish your interpretation of the work. Obviously this post simplifies, but it's already answered.

If you had to get permission, these millions of sound-alike tracks wouldn't exist. There's a compulsory license for cover songs, but the question becomes: is it still a cover if a machine does it? Or does it infringe the recordings' copyright since it's merely a "processed" version of the original recording? And how would you tell the difference between a human-cover and a machine-cover? Interesting times ahead...

Re: Blade Runner re-encoded using neural networks

#56

Post by radarsat1 »

Mithaldu wrote: post #46
radarsat1 wrote: post #45

Earlier quoted context omitted.

Right, but my point is that your numbers don't indicate anything about the actual potential as a compression format. It is certainly possible to use it that way, but there is still much that is uncertain about whether a neural network can do better than hand-crafted compression formats. So it's a little early to call it "next generation." It is literally meaningless to point out that "his neural network compresses ea…

> So it's a little early to call it "next generation." Please don't ignore words other people write, or at least reread before hitting post. I did say possibly . > I don't see why it's "impressive that it works at all." I thought it was implicit from my explanation that the "works at all" includes the fact that the data sizes are reasonable already. If we had the same result, but the intermediary data was 18 gigabyte…

> I thought it was implicit from my explanation that the "works at all" includes the fact that the data sizes are reasonable already.

Right, but why is that impressive if it doesn't actually result in a good reconstruction? I can take any collection of numbers and summarize it with the mean value, that doesn't imply that averaging is a good compression method.

If you consider this proof of concept, then what, exactly, concept does it prove? That statistics can represent a dataset?

Re: Blade Runner re-encoded using neural networks

#57

Post by wwweston »

buro9 wrote: post #25
svantana wrote: post #23

This article alludes to, but never actually brings up, something much more interesting than the project in question: How can copyright coexist with human-level automatic analysis and synthesis of works of art? For example, Spotify is flooded with "covers" of hit songs that are made to sound as similar to the original as possible. In my understanding, it doesn't matter how similar it sounds, as long as it's "remade",…

Music has that sorted already. There is copyright. There is publishing right. If you make a cover of a recording, it is the latter that you are involved with. You'd have to go to the publishers of a piece of music to license the right to publish your interpretation of the work. Obviously this post simplifies, but it's already answered.

There's some questions I don't think this addresses, though.

Right now, I have the ability to produce covers of lots of songs. That's addressed in law: if I make a recording, I need to get a mechanical license, if I'm performing at a certain scale I or the venue need to have an agreement with the right publisher for performance rights.

At a personal scale, though, I can perform covers in somebody's living room all day and nobody will care (or would be capable of doing much about it if they did). In fact, once I went looking for an agreement with a publisher for covers a small scale venue and they told me not to bother, because either the venue itself already had the agreement in place or it didn't matter.

But what happens if/when we everyone can have software that is capable of performing a cover of any work it's familiar with? Even if it doesn't have any particular digital copy directly encoded in any way we might now recognize?

If the law expands its definition of copyright to cover that situation, we're getting into territory in which it may be a copyright violation for me to remember a song. Or if enforcement of performance licensing laws starts to extend down to living-room scale performances, we're getting into really invasive surveillance.

But if that doesn't happen, those personal-performance-scale one-off covers (not stored, just part of the capabilities of the particular learning-machine) are probably going to be legal in the same way that my living room performances are right now.

Re: Blade Runner re-encoded using neural networks

#58

Post by rb2k_ »

svantana wrote: post #23

This article alludes to, but never actually brings up, something much more interesting than the project in question: How can copyright coexist with human-level automatic analysis and synthesis of works of art? For example, Spotify is flooded with "covers" of hit songs that are made to sound as similar to the original as possible. In my understanding, it doesn't matter how similar it sounds, as long as it's "remade",…

The question reminds me a bit of the idea behind https://en.wikipedia.org/wiki/OFFSystem

It was a fun project back in the day :)

Re: Blade Runner re-encoded using neural networks

#59

Post by tripzilch »

radarsat1 wrote: post #40
Mithaldu wrote: post #28

Earlier quoted context omitted.

This is possibly the next generation of video/audio/image codecs. What he did was create a specialized compression algorithm that works very well to compress the data that is each frame of Blade Runner, and decompress it (lossily, like mp3) back into a video stream. To put this into perspective: Blade Runner is 117 minutes long. At 25 frames per second, that is 175_500 frames. As he says, the input data he used was 2…

Except that in my understand, the decoded version looks like shit. So, while optimistic that this technology might be quite good eventually, more heuristics are probably needed for the right way to extract an optimal encoding. e.g. see the screenshots here: http://www.eteknix.com/blade-runner-gets-trippy-auto-encoded...

This network was trained on a single NVidia 960 GPU for two weeks, I wonder how much better it could have been with more computing power thrown at it?

Re: Blade Runner re-encoded using neural networks

#60

Post by blaze33 »

svantana wrote: post #23

This article alludes to, but never actually brings up, something much more interesting than the project in question: How can copyright coexist with human-level automatic analysis and synthesis of works of art? For example, Spotify is flooded with "covers" of hit songs that are made to sound as similar to the original as possible. In my understanding, it doesn't matter how similar it sounds, as long as it's "remade",…

Well that's a derivative work: https://en.wikipedia.org/wiki/Derivative_work
Post reply on HN