For Science: Does ZFS deduplication work on intros of TV shows?
manuelgrabowski.de
For Science: Does ZFS deduplication work on intros of TV shows?
1–10 of 55 posts
Re: For Science: Does ZFS deduplication work on intros of TV shows?
#2For shows with identical intros, mkv format supports linking to a separate file to store it. The separate file just has to be in the same directory.
Re: For Science: Does ZFS deduplication work on intros of TV shows?
#3For shows with identical intros, mkv format supports linking to a separate file to store it. The separate file just has to be in the same directory.
If you out your videos in a Quicktime .MOV file, you can put the intro in a separate file too, which can be placed anywhere (I think relative paths work too), you can even put a URL in there to load it from a server somewhere.
Alternately, you could decompress the intro frames - that might get ZFS dedup working, if the length of the file header is the same with all episodes.
Re: For Science: Does ZFS deduplication work on intros of TV shows?
#4For shows with identical intros, mkv format supports linking to a separate file to store it. The separate file just has to be in the same directory.
A feature which is frequently used by anime fansubbing groups, and almost never elsewhere.
Those groups often seem strangely bleeding-edge. They always adopt new features years before western groups will... I wonder why.
Re: For Science: Does ZFS deduplication work on intros of TV shows?
#5TV shows may have the same intro sequence, but the overlaid credits are always a bit different.
Hence any compression applied will always produce different results between episodes of a show. This would make any de-duplication extremely difficult.
Re: For Science: Does ZFS deduplication work on intros of TV shows?
#6I think this was a pretty dumb experiment, and the outcome was to be expected.
There will always be some sort of noise, pixels aligned differently etc., in a production like a tv series, and expecting the encoded output to be identical/matching on a blocklevel is pretty naive to say the least.
I did some experimenting using ZFS deduplication on MPEG2 files, where I encoded hundreds of MPEG2 dvd-sized videos, where 90% of the material was identical (the last 10% was affected by applying different watermarking techniques to the footage), and got some decent deduplication ratio (x1.2:1 or so).. But ZFS deduplication is expensive in memory/SSD, and it was definitely not worth it.
Re: For Science: Does ZFS deduplication work on intros of TV shows?
#7For shows with identical intros, mkv format supports linking to a separate file to store it. The separate file just has to be in the same directory.
A feature which is frequently used by anime fansubbing groups, and almost never elsewhere. Those groups often seem strangely bleeding-edge. They always adopt new features years before western groups will... I wonder why.
I'm guessing it's because they have to do much more work: western rip group just rip and upload, fansubs have to rip, translate (or find a translation), integrate subbing, recompress and upload. And they'll be going through each episode multiple times to QC timings, subs readability and check that subs doesn't prevent seeing anything of interest, at least for quality groups.
As a result they're always on the hunt for options which could help them do their stuff and:
* they'll find out about other possibly interesting features at the same time as they're poring over release notes
* because they might have to re-export their rips multiple times during QC, they can toy around extensively with features & settings (especially on early exports when they'll most likely have typos & al and will have to redo the export anyway)
tl;dr: they have plenty of opportunities to find and try out bleeding-edge features.
Re: For Science: Does ZFS deduplication work on intros of TV shows?
#8I think this was a pretty dumb experiment, and the outcome was to be expected. There will always be some sort of noise, pixels aligned differently etc., in a production like a tv series, and expecting the encoded output to be identical/matching on a blocklevel is pretty naive to say the least. I did some experimenting using ZFS deduplication on MPEG2 files, where I encoded hundreds of MPEG2 dvd-sized videos, where 90…
[deleted]
Re: For Science: Does ZFS deduplication work on intros of TV shows?
#9The reason I assumed deduplication wouldn't be effective on TV shows is because the intro isn't always at the same point as most shows have a prelude these days. Since the preludes are of different lengths, it means that certain things like key frames (if used on that particular codec) would appear on different frames of the intro sequence. Which makes me wonder how effective deduping would be on older TV shows where the intro is at the start of the programme.
As for compression being largely ineffective, the shows are already compressed anyway. In fact file system compression seems to be less relevant these days as most modern file formats have compression built in (even Office documents are just ZIP files). But as the author said, the overhead for compression isn't damaging for the performance* of ZFS - unlike with deduplication
* ZFS compression is particularly performant† when using the newer lz4 algorithm available in OpenZFS - which I'd highly recommend people using if they're not already.
† I know "performant" isn't technically a word. But it should be.
Re: For Science: Does ZFS deduplication work on intros of TV shows?
#10The reason I assumed deduplication wouldn't be effective on TV shows is because the intro isn't always at the same point as most shows have a prelude these days. Since the preludes are of different lengths, it means that certain things like key frames (if used on that particular codec) would appear on different frames of the intro sequence. Which makes me wonder how effective deduping would be on older TV shows where…
If people use and understand the word "performant", it's a word. That's how language works.