Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

For Science: Does ZFS deduplication work on intros of TV shows?

manuelgrabowski.de

Post reply on HN
Print view View original

41–50 of 55 posts

Re: For Science: Does ZFS deduplication work on intros of TV shows?

#41

Post by nitrogen »

bshimmin wrote: post #31

Perhaps I'm just in a bad mood today, but really, why must technical articles be festooned with meme GIFs featuring abusive language and children injuring themselves? What does it add? If your content is so boring to you, or to your prospective audience, that you think it needs livening up with that sort of garbage, maybe just don't write it in the first place.

Tastes vary, and what one finds offensive another might find hilarious. As long as we all get along in public, it shouldn't matter too much what makes us laugh when among friends.

Re: For Science: Does ZFS deduplication work on intros of TV shows?

#42

Post by VeejayRampay »

I've always dreamed that subbers would just provide one version of the credits in their season packs and provide a bash/mencoder/whatever script to glue the bits together after download. We're wasting so much bandwidth on opening and closing credits that people end up skipping anyway, it's crazy...

Re: For Science: Does ZFS deduplication work on intros of TV shows?

#43

Post by bshimmin »

matrixagent wrote: post #37
bshimmin wrote: post #31

Perhaps I'm just in a bad mood today, but really, why must technical articles be festooned with meme GIFs featuring abusive language and children injuring themselves? What does it add? If your content is so boring to you, or to your prospective audience, that you think it needs livening up with that sort of garbage, maybe just don't write it in the first place.

I tend to agree with you, but the article wasn't supposed to be very technical in the first place, and: both those GIFs are from Dexter, the show of which I used a season in my tests. It's not completely unrelated.

I didn't realise they were from Dexter, though I stand by what I said. Clearly the masses on HN agree with your choices, though, since I'm getting down-voted!

I'm writing some technical documentation for a project right now - perhaps I'll put in a few tasteful images from Baywatch to make sure the project managers who have to read it will make it to the end.

Re: For Science: Does ZFS deduplication work on intros of TV shows?

#44

Post by TheCondor »

ccrush wrote: post #28
josteink wrote: post #25

If you were going to do this for science, a good start would be to analyze the intro of shows in question, using rudimentary tools such as md5sum on a per-block basis. You would then quickly discover that the chances of huge gains would be small, without ever even looking into the ZFS aspects.

ZFS supports byte-level deduplication where it can compute anchor points so that it works on unaligned data. However, quite a few a few assumptions about the entropy of the data set could have been inferred from the "iTunes TV Show" file choice, which clearly means the files would be compressed and encrypted.

Is that new? Zfs has supported block deduplication for a few years but I'm unaware if it supporting byte level deduplication.

Re: For Science: Does ZFS deduplication work on intros of TV shows?

#45

Post by ccrush »

matrixagent wrote: post #32
ccrush wrote: post #24

Yes, ZFS deduplication is not very useful when trying to deduplicate encrypted and compressed video files. Encryption is supposed to make the contents of the file look random. Compression is itself a form of in-file deduplication. Where ZFS deduplication will be extremely useful: on a SAN in a video editing studio where originals are kept in raw format. Even then, file-level and block-level dedups will probably be th…

The files are not encrypted. I mention the aspect of doing it with files that are already compressed and that I never expected it to work for that very reason. I just wanted to to for once actually do it and I don't see why you need to get so mad about that.

Unless you stripped the DRM from the files, yes, they are encrypted. At least according to the Wikipedia page on FairPlay [http://en.wikipedia.org/wiki/FairPlay], all the iTunes files are encrypted.

Also, wouldn't proving it doesn't work by setting up a deliberately flawed scenario amount to a bit of blogsturbation? Should I write an article about "finding out whether my computer will turn on even if it's unplugged"? Answer: no. But read my article about me TRYING it just to be sure.

I think you should write a follow-up article where you splice together raw video files and include a similar segment in all the different files. Then, put that through the dedup test. I would actually be interested to see the ability of ZFS implementations to find worthwhile anchor points within files and do smart byte-level deduplication.

Re: For Science: Does ZFS deduplication work on intros of TV shows?

#46

Post by Leynos »

AlyssaRowan wrote: post #38
Filligree wrote: post #4

Earlier quoted context omitted.

A feature which is frequently used by anime fansubbing groups, and almost never elsewhere. Those groups often seem strangely bleeding-edge. They always adopt new features years before western groups will... I wonder why.

Indeed: they were early adopters of many things: Ogg, Vorbis, Matroska (and the advanced features like linked ordered chapters which is designed for the use-case in OP's scenario), h.264 High10 Profile, advanced animated subtitles formats... I think the long answer has something to do with what they do benefitting from it more than other things, and the short answer is "because they're geeks"; so they perhaps care le…

They also don't have releasing standards to adhere to. As was mentioned earlier, most fansub viewers are watching on their computers, which can cope with newer codecs, subtitle formats and container features with ease, where-as western TV rips are targeted at viewers using lowest common denominator hardware (and encoded according to a set of standards agreed between groups).

Changes to those standards can cause significant upheaval, as happened when the standard def TV groups multilaterally agreed to switch from XviD in AVI to h.264 in MP4 container.

AIUI, the tertiary anime groups, who take h.264/MKV/ASS releases and re-encode as MP4 with burned in subtitles, cater to those using lower spec hardware but who still want high def video.

Re: For Science: Does ZFS deduplication work on intros of TV shows?

#47

Post by ccrush »

TheCondor wrote: post #44
ccrush wrote: post #28

Earlier quoted context omitted.

ZFS supports byte-level deduplication where it can compute anchor points so that it works on unaligned data. However, quite a few a few assumptions about the entropy of the data set could have been inferred from the "iTunes TV Show" file choice, which clearly means the files would be compressed and encrypted.

Is that new? Zfs has supported block deduplication for a few years but I'm unaware if it supporting byte level deduplication.

https://blogs.oracle.com/bonwick/entry/zfs_dedup

Re: For Science: Does ZFS deduplication work on intros of TV shows?

#48

Post by theoh »

matrixagent wrote: post #32
ccrush wrote: post #24

Yes, ZFS deduplication is not very useful when trying to deduplicate encrypted and compressed video files. Encryption is supposed to make the contents of the file look random. Compression is itself a form of in-file deduplication. Where ZFS deduplication will be extremely useful: on a SAN in a video editing studio where originals are kept in raw format. Even then, file-level and block-level dedups will probably be th…

The files are not encrypted. I mention the aspect of doing it with files that are already compressed and that I never expected it to work for that very reason. I just wanted to to for once actually do it and I don't see why you need to get so mad about that.

I can see where he's coming from. Any differences in the data caused by encryption or compression of the final video files don't even come into play here unless the raw video data they are produced from is bit-for-bit identical from one episode to the next, say in a precise lossless format like CCIR 601. Maybe this is true at the production company but I wouldn't be surprised if it isn't. It would actually be somewhat interesting to know what the typical workflow at the production company is for gluing the title sequence together with the rest of the show. Your blog post doesn't do enough to clarify these issues.

Re: For Science: Does ZFS deduplication work on intros of TV shows?

#49

Post by ryao »

ccrush wrote: post #28
josteink wrote: post #25

If you were going to do this for science, a good start would be to analyze the intro of shows in question, using rudimentary tools such as md5sum on a per-block basis. You would then quickly discover that the chances of huge gains would be small, without ever even looking into the ZFS aspects.

ZFS supports byte-level deduplication where it can compute anchor points so that it works on unaligned data. However, quite a few a few assumptions about the entropy of the data set could have been inferred from the "iTunes TV Show" file choice, which clearly means the files would be compressed and encrypted.

This is incorrect. Byte-level deduplication does work with anchor points, but it is extremely expensive. Jeff decided to let the application do byte-level deduplication and implement block-level deduplication instead. This gave us something that was reasonable to implement at the filesystem level.

Re: For Science: Does ZFS deduplication work on intros of TV shows?

#50

Post by a-dub »

gilgoomesh wrote: post #35

I challenge you to record 30 seconds of a fixed test pattern from TV on two separate occasions and have it encode to identical bits. I don't think it can be done. Many media formats (including all digital TV formats) encode a rolling hardware timestamp of 33 bits (or more) which won't be the same for two separate segments by random chance. Audio, video, subtitles and other metadata will be ordered differently in the…

Could make for an interesting codec idea though! A codebook based compressor that works on corpora of audio/video seems doable. More interesting would be a codec that aims not only to exploit models of human perception to throw away bits, but also exploits those models to produce "canonical" compressed forms that would be suitable for byte/block level deduplication. Of course, interesting things are hard, and errors would be hilarious. (See: Xerox and their copiers that change letters and numbers in documents...)
Post reply on HN