A codec that could search for reference frames from large collections of media might be pretty interesting for some applications, especially ones with large storage demands and repetitive material. The indexing system would be an interesting problem since it would have to be very low overhead. Wonder if any systems like this exist?
For Science: Does ZFS deduplication work on intros of TV shows?
21–30 of 55 posts
Re: For Science: Does ZFS deduplication work on intros of TV shows?
#22A few bits of change in uncompressed video can change the following stream in the compressed video completely. Deduplication isn't intelligent enough to understand "video" and compress for that -- that's what video codecs are for. So it's not surprising that deduplication would not prove useful for compressed video. You could see the advantage in uncompressed streams.
Re: For Science: Does ZFS deduplication work on intros of TV shows?
#23I am curious and did not read the docs: Does ZFS deduplication work on parts of files? I would have expected that filesystem deduplication just deduplicate complete files (for conveniance). Deduplication of file parts could of course be good for some types of files, with a fixed structure and with no "noise" -- but with videos, there is no chance I guess without manually linking parts.
Re: For Science: Does ZFS deduplication work on intros of TV shows?
#24Yes, ZFS deduplication is not very useful when trying to deduplicate encrypted and compressed video files. Encryption is supposed to make the contents of the file look random. Compression is itself a form of in-file deduplication. Where ZFS deduplication will be extremely useful: on a SAN in a video editing studio where originals are kept in raw format. Even then, file-level and block-level dedups will probably be the least effective means of deduplication. What we need is byte-level deduplication which computes anchor points from which data ought to be deduplicated. Even then, it sounds like the dedup feature would be scripted so that it be a part of the tooling, and would be updated with the team's projects. I would go as far as to say the author chose a deliberately eye-catching headline and coupled up a few Linux commands with some "hilarious" GIFs to address the wannabe geeks out there while displaying a staggering pile of shamelessness to even the remotely competent of his blog's visitors.
Re: For Science: Does ZFS deduplication work on intros of TV shows?
#25If you were going to do this for science, a good start would be to analyze the intro of shows in question, using rudimentary tools such as md5sum on a per-block basis.
You would then quickly discover that the chances of huge gains would be small, without ever even looking into the ZFS aspects.
Re: For Science: Does ZFS deduplication work on intros of TV shows?
#26Earlier quoted context omitted.
A feature which is frequently used by anime fansubbing groups, and almost never elsewhere. Those groups often seem strangely bleeding-edge. They always adopt new features years before western groups will... I wonder why.
I'm guessing it's because they have to do much more work: western rip group just rip and upload, fansubs have to rip, translate (or find a translation), integrate subbing, recompress and upload. And they'll be going through each episode multiple times to QC timings, subs readability and check that subs doesn't prevent seeing anything of interest, at least for quality groups. As a result they're always on the hunt for…
Also, they target much more tech-savvy people who run the latest media player with the latest codec on the latest OS. There is much more chance that the evolutions in the material is readable by consumers; there is less backward-compatibility to support.
Re: For Science: Does ZFS deduplication work on intros of TV shows?
#27Since videos from the iTunes Store are encrypted, it seems like the individual files should look like random noise to the filesystem anyway. Unless there is something I don't understand happening.
There are ways to remove the DRM. In case you don't want to rely on Apple keeping their activation servers online forever.
I don't think OP did it in the post, so I'm also inclined to believe that he did not see deduplication in part because of that.
Re: For Science: Does ZFS deduplication work on intros of TV shows?
#28If you were going to do this for science, a good start would be to analyze the intro of shows in question, using rudimentary tools such as md5sum on a per-block basis. You would then quickly discover that the chances of huge gains would be small, without ever even looking into the ZFS aspects.
ZFS supports byte-level deduplication where it can compute anchor points so that it works on unaligned data. However, quite a few a few assumptions about the entropy of the data set could have been inferred from the "iTunes TV Show" file choice, which clearly means the files would be compressed and encrypted.
Re: For Science: Does ZFS deduplication work on intros of TV shows?
#29Earlier quoted context omitted.
There are ways to remove the DRM. In case you don't want to rely on Apple keeping their activation servers online forever.
I don't think OP did it in the post, so I'm also inclined to believe that he did not see deduplication in part because of that.
dewey is correct, DRM/encryption is not the issue.
Re: For Science: Does ZFS deduplication work on intros of TV shows?
#30Reminds me of the DVD sets of Sailor Moon I made years ago where I spliced the same video sequences in for the intro and outro. I never got so far as deduplicating the henshi sequences because there often are different voice-overs during them.