For Science: Does ZFS deduplication work on intros of TV shows? • Hacker News Board

Re: For Science: Does ZFS deduplication work on intros of TV shows?

#51

Post by Kayou » Mon, Sep 29, 2014, 3:09 PM UTC

gilgoomesh wrote: post #35

I challenge you to record 30 seconds of a fixed test pattern from TV on two separate occasions and have it encode to identical bits. I don't think it can be done. Many media formats (including all digital TV formats) encode a rolling hardware timestamp of 33 bits (or more) which won't be the same for two separate segments by random chance. Audio, video, subtitles and other metadata will be ordered differently in the…

I agree for TV, but what about the case of the author, where he gets the episode from iTunes? Blu-Ray and iTunes episodes have a chance to have the intro timed perfectly the same as they are probably exported from a kind-of video editing workflow where they copy-paste the intro. At least, that's what I imagine. So the intro in their workflow is probably the same (I mean, the same video file), but somehow the export to h264 makes it different for each episode. Maybe the psy optimizations use dithering or another trick involving randomization.

Re: For Science: Does ZFS deduplication work on intros of TV shows?

#52

Post by laumars » Mon, Sep 29, 2014, 3:10 PM UTC

PaulHoule wrote: post #34

laumars wrote: post #9

The reason I assumed deduplication wouldn't be effective on TV shows is because the intro isn't always at the same point as most shows have a prelude these days. Since the preludes are of different lengths, it means that certain things like key frames (if used on that particular codec) would appear on different frames of the intro sequence. Which makes me wonder how effective deduping would be on older TV shows where…

The semantic problem I see with "performant", which is sharpened by the word transformation, is that there almost always multiple dimensions of performance. For instance, "performance" in terms of time is relevant to all IT systems, but there is usually some other dimension, for instance, compression ratio or accuracy, that matters too. Thus, "fast" is a better adjective when it applies because it is more clear.

In this instance, all of the above. It's faster and offers higher compression ratios LZJB: http://wiki.illumos.org/display/illumos/LZ4+Compression

As for accuracy, well you'd expect any and all deflation compression to be 100% accurate anyway. Anything less than 100% would corrupt your data (unlike with audio / video "lossy" compression where you can remove / group non-perceivable data)

Re: For Science: Does ZFS deduplication work on intros of TV shows?

#53

Post by Kayou » Mon, Sep 29, 2014, 3:13 PM UTC

a-dub wrote: post #50

gilgoomesh wrote: post #35

I challenge you to record 30 seconds of a fixed test pattern from TV on two separate occasions and have it encode to identical bits. I don't think it can be done. Many media formats (including all digital TV formats) encode a rolling hardware timestamp of 33 bits (or more) which won't be the same for two separate segments by random chance. Audio, video, subtitles and other metadata will be ordered differently in the…

Could make for an interesting codec idea though! A codebook based compressor that works on corpora of audio/video seems doable. More interesting would be a codec that aims not only to exploit models of human perception to throw away bits, but also exploits those models to produce "canonical" compressed forms that would be suitable for byte/block level deduplication. Of course, interesting things are hard, and errors…

Also, I don't know if this is already implemented in some codec, but it would be interesting to re-use key frames from previous shots. For instance when you have a dialog and the camera switches from one person to another, a conventional codec probably creates a new i-frame for each shot whereas an intelligent codec would recognize that this is the continuation of the previous shot.

Re: For Science: Does ZFS deduplication work on intros of TV shows?

#54

Post by Kayou » Mon, Sep 29, 2014, 3:20 PM UTC

AlyssaRowan wrote: post #38

Filligree wrote: post #4

Earlier quoted context omitted.

A feature which is frequently used by anime fansubbing groups, and almost never elsewhere. Those groups often seem strangely bleeding-edge. They always adopt new features years before western groups will... I wonder why.

Indeed: they were early adopters of many things: Ogg, Vorbis, Matroska (and the advanced features like linked ordered chapters which is designed for the use-case in OP's scenario), h.264 High10 Profile, advanced animated subtitles formats... I think the long answer has something to do with what they do benefitting from it more than other things, and the short answer is "because they're geeks"; so they perhaps care le…

I think that Anime release groups are also a lot less "regulated" than US TV rip scene. For animes, you can just ask your friends to help you translate an uncovered anime in your country, you get the "raw" from japan and you sub. For TV, they have rules as complicated as this: http://scenerules.irc.gs/n.html?id=2011_TV_X264_u2.nfo if they don't follow them, they're excluded from the scene.

Re: For Science: Does ZFS deduplication work on intros of TV shows?

#55

Post by lkateley » Mon, Sep 29, 2014, 4:28 PM UTC

ccrush wrote: post #47

TheCondor wrote: post #44

Earlier quoted context omitted.

Is that new? Zfs has supported block deduplication for a few years but I'm unaware if it supporting byte level deduplication.

https://blogs.oracle.com/bonwick/entry/zfs_dedup

Not new.. dated 2009.

Oracle seems to be doing alot of the lately where they are posting really old collaterial