Perhaps I'm just in a bad mood today, but really, why must technical articles be festooned with meme GIFs featuring abusive language and children injuring themselves? What does it add? If your content is so boring to you, or to your prospective audience, that you think it needs livening up with that sort of garbage, maybe just don't write it in the first place.
For Science: Does ZFS deduplication work on intros of TV shows?
31–40 of 55 posts
Re: For Science: Does ZFS deduplication work on intros of TV shows?
#32Yes, ZFS deduplication is not very useful when trying to deduplicate encrypted and compressed video files. Encryption is supposed to make the contents of the file look random. Compression is itself a form of in-file deduplication. Where ZFS deduplication will be extremely useful: on a SAN in a video editing studio where originals are kept in raw format. Even then, file-level and block-level dedups will probably be th…
The files are not encrypted. I mention the aspect of doing it with files that are already compressed and that I never expected it to work for that very reason. I just wanted to to for once actually do it and I don't see why you need to get so mad about that.
Re: For Science: Does ZFS deduplication work on intros of TV shows?
#33I am curious and did not read the docs: Does ZFS deduplication work on parts of files? I would have expected that filesystem deduplication just deduplicate complete files (for conveniance). Deduplication of file parts could of course be good for some types of files, with a fixed structure and with no "noise" -- but with videos, there is no chance I guess without manually linking parts.
Yep, it does. http://arstechnica.com/information-technology/2014/01/bitrot... has a pretty nice overview.
Re: For Science: Does ZFS deduplication work on intros of TV shows?
#34The reason I assumed deduplication wouldn't be effective on TV shows is because the intro isn't always at the same point as most shows have a prelude these days. Since the preludes are of different lengths, it means that certain things like key frames (if used on that particular codec) would appear on different frames of the intro sequence. Which makes me wonder how effective deduping would be on older TV shows where…
The semantic problem I see with "performant", which is sharpened by the word transformation, is that there almost always multiple dimensions of performance.
For instance, "performance" in terms of time is relevant to all IT systems, but there is usually some other dimension, for instance, compression ratio or accuracy, that matters too.
Thus, "fast" is a better adjective when it applies because it is more clear.
Re: For Science: Does ZFS deduplication work on intros of TV shows?
#35I challenge you to record 30 seconds of a fixed test pattern from TV on two separate occasions and have it encode to identical bits. I don't think it can be done.
Many media formats (including all digital TV formats) encode a rolling hardware timestamp of 33 bits (or more) which won't be the same for two separate segments by random chance. Audio, video, subtitles and other metadata will be ordered differently in the stream because they all comes from sources that have their own separate clocks. Synchronization between clocks at different stages in the media pipeline will cause different frames in the sequence to be dropped, padded, made into keyframes, etc, which then affects every bit in subsequent frames. TV stations use time-based watermarks. TV stations use digital compositing software that may retain bits from previous frames on an ongoing basis. Many studio media pipelines still involve analog steps.
The list of complications goes on.
Unless your pipeline is lossless, uses only digital sources and uses totally synchronized clocks for all stages, you're about as likely to get an MD5 hash collision by accident as you are to get any two non-trivial sequences of compressed video to be bit-identical.
Re: For Science: Does ZFS deduplication work on intros of TV shows?
#36I think this was a pretty dumb experiment, and the outcome was to be expected. There will always be some sort of noise, pixels aligned differently etc., in a production like a tv series, and expecting the encoded output to be identical/matching on a blocklevel is pretty naive to say the least. I did some experimenting using ZFS deduplication on MPEG2 files, where I encoded hundreds of MPEG2 dvd-sized videos, where 90…
I guess you missed the part where I say almost the exact same thing as you just did, i.e. that I did never expect it to work?
Re: For Science: Does ZFS deduplication work on intros of TV shows?
#37Perhaps I'm just in a bad mood today, but really, why must technical articles be festooned with meme GIFs featuring abusive language and children injuring themselves? What does it add? If your content is so boring to you, or to your prospective audience, that you think it needs livening up with that sort of garbage, maybe just don't write it in the first place.
I tend to agree with you, but the article wasn't supposed to be very technical in the first place, and: both those GIFs are from Dexter, the show of which I used a season in my tests. It's not completely unrelated.
Re: For Science: Does ZFS deduplication work on intros of TV shows?
#38For shows with identical intros, mkv format supports linking to a separate file to store it. The separate file just has to be in the same directory.
A feature which is frequently used by anime fansubbing groups, and almost never elsewhere. Those groups often seem strangely bleeding-edge. They always adopt new features years before western groups will... I wonder why.
Indeed: they were early adopters of many things: Ogg, Vorbis, Matroska (and the advanced features like linked ordered chapters which is designed for the use-case in OP's scenario), h.264 High10 Profile, advanced animated subtitles formats...
I think the long answer has something to do with what they do benefitting from it more than other things, and the short answer is "because they're geeks"; so they perhaps care less about slow hardware adoption of things like 10-bit than other encoders.
Re: For Science: Does ZFS deduplication work on intros of TV shows?
#39TV shows may have the same intro sequence, but the overlaid credits are always a bit different. Hence any compression applied will always produce different results between episodes of a show. This would make any de-duplication extremely difficult.
I have to admit that I had not checked that, but I think Dexter's intro is really the same and the changing cast lists appear during the first scenes afterwards. But for most shows you're right, that would be another reason why it could never work.
Re: For Science: Does ZFS deduplication work on intros of TV shows?
#40A codec that could search for reference frames from large collections of media might be pretty interesting for some applications, especially ones with large storage demands and repetitive material. The indexing system would be an interesting problem since it would have to be very low overhead. Wonder if any systems like this exist?
The player would have to be able to reference a huge number of sparsely-positioned reference frames.
The seeking might make it impractical.