Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Peeking Inside Gigantic Zips with Only Kilobytes

ritiksahni.com

Post reply on HN
Print view View original

21–30 of 30 posts

Re: Peeking Inside Gigantic Zips with Only Kilobytes

#21

Post by nicman23 »

Lammy wrote: post #18
duskwuff wrote: post #17

Earlier quoted context omitted.

ZIP isn't a terrible format, but it has a couple of flaws and limitations which make it a less than ideal format for long-term archiving. The biggest ones I'd call out are: 1) The format has limited and archaic support for file metadata - e.g. file modification times are stored as a MS-DOS timestamp with a 2-second (!) resolution, and there's no standard system for representing other metadata. 2) The single-level cen…

> there's no straightforward way to support "solid" compression. I do it by ignoring ZIP's native compression entirely, using store-only ZIP files and then compressing the whole thing at the filesystem level instead. Here's an example comparison of the same WWW site rip in a DEFLATE ZIP, in a store-only ZIP with zstd filesystem compression, in a tar with same zstd filesystem compression (identical size but less usefu…

lool why use zip then anyways? put them in a folder

Re: Peeking Inside Gigantic Zips with Only Kilobytes

#22

Post by Lammy »

nicman23 wrote: post #21
Lammy wrote: post #18

Earlier quoted context omitted.

> there's no straightforward way to support "solid" compression. I do it by ignoring ZIP's native compression entirely, using store-only ZIP files and then compressing the whole thing at the filesystem level instead. Here's an example comparison of the same WWW site rip in a DEFLATE ZIP, in a store-only ZIP with zstd filesystem compression, in a tar with same zstd filesystem compression (identical size but less usefu…

lool why use zip then anyways? put them in a folder

It's for when you have a very large number of mostly-identical files, like web pages with consistent header and footer. If 408MiB versus 3.8GiB is a meaningless difference to you then sure don't bother with compression, but why I want it should be very obvious to most people here.

Re: Peeking Inside Gigantic Zips with Only Kilobytes

#23

Post by gildas »

duskwuff wrote: post #17
toomuchtodo wrote: post #8

Earlier quoted context omitted.

Based on your experience, is zip the optimal archive format for long term digital archival in object storage if the use case calls for reading archives via http for scanning and cherry picking? Or is there a more optimal archive format?

ZIP isn't a terrible format, but it has a couple of flaws and limitations which make it a less than ideal format for long-term archiving. The biggest ones I'd call out are: 1) The format has limited and archaic support for file metadata - e.g. file modification times are stored as a MS-DOS timestamp with a 2-second (!) resolution, and there's no standard system for representing other metadata. 2) The single-level cen…

FYI, zip.js has no issues with 1 (it can be fixed with standard extra fields), 3 (zip64 support), and 5 (you cannot have more than 64K of comment data at the end of the file).

Re: Peeking Inside Gigantic Zips with Only Kilobytes

#24

Post by duskwuff »

gildas wrote: post #23
duskwuff wrote: post #17

Earlier quoted context omitted.

ZIP isn't a terrible format, but it has a couple of flaws and limitations which make it a less than ideal format for long-term archiving. The biggest ones I'd call out are: 1) The format has limited and archaic support for file metadata - e.g. file modification times are stored as a MS-DOS timestamp with a 2-second (!) resolution, and there's no standard system for representing other metadata. 2) The single-level cen…

FYI, zip.js has no issues with 1 (it can be fixed with standard extra fields), 3 (zip64 support), and 5 (you cannot have more than 64K of comment data at the end of the file).

With regard for the first two - that's good for zip.js, but the problem is that support for those features isn't universal. There's been a lot of fragmentation over the last 36 years (!).

As far as the last (file type detection) goes, the generally agreed upon standard is that file formats should be "sniffable" by looking for a signature in the file's header - ideally within the first few bytes of the file. Having to search through 64 KB of the file's end for a signature is a major departure from that pattern.

Re: Peeking Inside Gigantic Zips with Only Kilobytes

#25

Post by nicman23 »

Lammy wrote: post #22
nicman23 wrote: post #21

Earlier quoted context omitted.

lool why use zip then anyways? put them in a folder

It's for when you have a very large number of mostly-identical files, like web pages with consistent header and footer. If 408MiB versus 3.8GiB is a meaningless difference to you then sure don't bother with compression, but why I want it should be very obvious to most people here.

you completely missed what i asked you but ok

Re: Peeking Inside Gigantic Zips with Only Kilobytes

#26

Post by rtk0 »

dabinat wrote: post #15

I wrote a Rust command-line tool to do this for internal use in my SaaS. The motivation was to be able to index the contents of zip files stored on S3 without incurring significant egress charges. Is this something that people would generally find useful if it was open-sourced?

Yes, the motivation to explore was something similar. I was curious if downloading ZIP files could be made more efficient over the web.

Re: Peeking Inside Gigantic Zips with Only Kilobytes

#27

Post by rtk0 »

saulpw wrote: post #19

Here's my Python library that does the same[0]. And it's incorporated into VisiData so you can view a .csv from within a .zip file over HTTP without downloading the whole .zip file. [0] https://github.com/saulpw/unzip-http/

Lovely! Thanks for sharing. I had so much fun learning about ZIP and writing the blog post.

Re: Peeking Inside Gigantic Zips with Only Kilobytes

#29

Post by Lammy »

nicman23 wrote: post #25
Lammy wrote: post #22

Earlier quoted context omitted.

It's for when you have a very large number of mostly-identical files, like web pages with consistent header and footer. If 408MiB versus 3.8GiB is a meaningless difference to you then sure don't bother with compression, but why I want it should be very obvious to most people here.

you completely missed what i asked you but ok

I don't think I did, but please explain :)

The last example in my list of four file sizes is them in a folder. Filesystem compression works at the file level, so you have to turn many-almost-identical-files into one file in order to benefit from it. ZFS does have block-level deduplication, but that's it's own can of worms that shouldn't be turned on flippantly due to resource requirements and `recordsize` tuning needed to really benefit from it.

Re: Peeking Inside Gigantic Zips with Only Kilobytes

#30

Post by nicman23 »

Lammy wrote: post #29
nicman23 wrote: post #25

Earlier quoted context omitted.

you completely missed what i asked you but ok

I don't think I did, but please explain :) The last example in my list of four file sizes is them in a folder. Filesystem compression works at the file level, so you have to turn many-almost-identical-files into one file in order to benefit from it. ZFS does have block-level deduplication, but that's it's own can of worms that shouldn't be turned on flippantly due to resource requirements and `recordsize` tuning need…

you do not need dedup just use reflinks for everything. if that workflow does not work then eh i understand why you would use zips

although zfs dedup is probably better in 2025

Post reply on HN