Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

For Science: Does ZFS deduplication work on intros of TV shows?

manuelgrabowski.de

Post reply on HN
Print view View original

11–20 of 55 posts

Re: For Science: Does ZFS deduplication work on intros of TV shows?

#11

Post by aukaost »

zimpenfish wrote: post #10
laumars wrote: post #9

The reason I assumed deduplication wouldn't be effective on TV shows is because the intro isn't always at the same point as most shows have a prelude these days. Since the preludes are of different lengths, it means that certain things like key frames (if used on that particular codec) would appear on different frames of the intro sequence. Which makes me wonder how effective deduping would be on older TV shows where…

If people use and understand the word "performant", it's a word. That's how language works.

It is a word in German. Went to check if the author might be German - and he is. http://dict.leo.org/#/search=performant&searchLoc=0&resultOr...

Re: For Science: Does ZFS deduplication work on intros of TV shows?

#12

Post by laumars »

zimpenfish wrote: post #10
laumars wrote: post #9

The reason I assumed deduplication wouldn't be effective on TV shows is because the intro isn't always at the same point as most shows have a prelude these days. Since the preludes are of different lengths, it means that certain things like key frames (if used on that particular codec) would appear on different frames of the intro sequence. Which makes me wonder how effective deduping would be on older TV shows where…

If people use and understand the word "performant", it's a word. That's how language works.

In the evolutionary sense languages, I agree. But there's also a lot of people who strictly disagree with slang, "verbing" and other colloquialisms - instead believing language should be applied strictly (people otherwise known as "language Nazis").

My "performant" footnote was just an attempt to pre-empt such attacks; though ironically including the footnote has now sparked the tangent itself.

Re: For Science: Does ZFS deduplication work on intros of TV shows?

#13

Post by laumars »

bio4m wrote: post #5

TV shows may have the same intro sequence, but the overlaid credits are always a bit different. Hence any compression applied will always produce different results between episodes of a show. This would make any de-duplication extremely difficult.

I think it's a bit of a stretch to say the opening credits are "always a bit different". I can think of quite a few shows display the episode directors and any guest stars after the intro (ie during the first scene of the main episode). I presume they do this to save editing time and cost changing the intro - though there's also the convenient side effect of still having the credited names appear on screen even when viewers fast-forward through the intro.

Re: For Science: Does ZFS deduplication work on intros of TV shows?

#14

Post by rplnt »

masklinn wrote: post #7
Filligree wrote: post #4

Earlier quoted context omitted.

A feature which is frequently used by anime fansubbing groups, and almost never elsewhere. Those groups often seem strangely bleeding-edge. They always adopt new features years before western groups will... I wonder why.

I'm guessing it's because they have to do much more work: western rip group just rip and upload, fansubs have to rip, translate (or find a translation), integrate subbing, recompress and upload. And they'll be going through each episode multiple times to QC timings, subs readability and check that subs doesn't prevent seeing anything of interest, at least for quality groups. As a result they're always on the hunt for…

I only wish "western" releases were already packed with subtitles. Especially now that opensubtitles tries to sell ad space directly in the subtitles.

Re: For Science: Does ZFS deduplication work on intros of TV shows?

#16

Post by jamesbrownuhh »

There's basically no way to make this work because, as has been mentioned, in compressed formats there's every chance that title sequences will occur at different times in the show, so will occur at different points in the group-of-pictures sequences in compressed media.

And especially for rips from an already-compressed signal source (like TV), the odds are well against a sequence, although identical visually, encoding into exactly the same sequence of bits digitally, as there are so many factors that can vary the datastream you receive, even before it's ripped and re-encoded.

To stand any chance of this working, you'd really need to be storing and comparing uncompressed frames direct from the master, before ANY kind of variable encoding or compression. But if you have enough storage to be working with files on that scale, deduplication is probably not your biggest concern. :)

Re: For Science: Does ZFS deduplication work on intros of TV shows?

#17

Post by dewey »

oldmanjay wrote: post #15

Since videos from the iTunes Store are encrypted, it seems like the individual files should look like random noise to the filesystem anyway. Unless there is something I don't understand happening.

There are ways to remove the DRM. In case you don't want to rely on Apple keeping their activation servers online forever.

Re: For Science: Does ZFS deduplication work on intros of TV shows?

#18

Post by ZoFreX »

laumars wrote: post #12
zimpenfish wrote: post #10

Earlier quoted context omitted.

If people use and understand the word "performant", it's a word. That's how language works.

In the evolutionary sense languages, I agree. But there's also a lot of people who strictly disagree with slang, "verbing" and other colloquialisms - instead believing language should be applied strictly (people otherwise known as "language Nazis"). My "performant" footnote was just an attempt to pre-empt such attacks; though ironically including the footnote has now sparked the tangent itself.

Relevant Calvin & Hobbes: http://www.gocomics.com/calvinandhobbes/1993/01/25

Re: For Science: Does ZFS deduplication work on intros of TV shows?

#19

Post by ZoFreX »

kapsel wrote: post #6

I think this was a pretty dumb experiment, and the outcome was to be expected. There will always be some sort of noise, pixels aligned differently etc., in a production like a tv series, and expecting the encoded output to be identical/matching on a blocklevel is pretty naive to say the least. I did some experimenting using ZFS deduplication on MPEG2 files, where I encoded hundreds of MPEG2 dvd-sized videos, where 90…

It may be a "dumb experiment" if you already understand how the underlying systems (both the file system, and video encoding) work - but the author did not, and now having done the experiment, they understand more about them.

Re: For Science: Does ZFS deduplication work on intros of TV shows?

#20

Post by zimpenfish »

laumars wrote: post #12
zimpenfish wrote: post #10

Earlier quoted context omitted.

If people use and understand the word "performant", it's a word. That's how language works.

In the evolutionary sense languages, I agree. But there's also a lot of people who strictly disagree with slang, "verbing" and other colloquialisms - instead believing language should be applied strictly (people otherwise known as "language Nazis"). My "performant" footnote was just an attempt to pre-empt such attacks; though ironically including the footnote has now sparked the tangent itself.

It's a fun thing to point out to those people how many of the words they're using to lambast you have actually changed meaning or are new words. They get -very- flustered and sulky.
Post reply on HN