Peeking Inside Gigantic Zips with Only Kilobytes

#1

Post by rtk0 » Sun, Oct 12, 2025, 9:57 AM UTC

Peeking Inside Gigantic Zips with Only Kilobytes
ritiksahni.com

Re: Peeking Inside Gigantic Zips with Only Kilobytes

#2

Post by rtk0 » Sun, Oct 12, 2025, 9:57 AM UTC

In this blog, I wrote about the architecture of a ZIP file and how we can leverage HTTP range requests to download files without decompressing the archive, in-browser.

Re: Peeking Inside Gigantic Zips with Only Kilobytes

#3

Post by gildas » Thu, Oct 16, 2025, 7:51 PM UTC

For implementation in a library, you can use HttpRangeReader [1][2] in zip.js [3] (disclaimer: I am the author). It's a solid feature that has been in the library for about 10 years.

[1] https://gildas-lormeau.github.io/zip.js/api/classes/HttpRang...

[2] https://github.com/gildas-lormeau/zip.js/blob/master/tests/a...

[3] https://github.com/gildas-lormeau/zip.js

Re: Peeking Inside Gigantic Zips with Only Kilobytes

#4

Post by xg15 » Thu, Oct 16, 2025, 7:55 PM UTC

This is really cool! Could also make a useful standalone command line tool.

I think the general pattern - using the range header + prior knowledge of a file format to only download the parts of a file that are relevant - is still really underutilized.

One small problem I see is that a server that does not support range requests would just try to send you the entire file in the first request, I think.

So maybe doing a preflight HEAD request first to see if the server sends back Accept-Ranges could be useful.

https://developer.mozilla.org/en-US/docs/Web/HTTP/Guides/Ran...

Re: Peeking Inside Gigantic Zips with Only Kilobytes

#5

Post by jeffrallen » Thu, Oct 16, 2025, 8:04 PM UTC

Here's the results of my investigation into the same question:

https://blog.nella.org/2016/01/17/seeking-http/

(Originally written for Advent of Go.)

Re: Peeking Inside Gigantic Zips with Only Kilobytes

#6

Post by HPsquared » Thu, Oct 16, 2025, 8:10 PM UTC

7-zip does this. You can see it if you open (to view) a large ZIP file on slow network drive. There's no way it is downloading the whole thing. You can extract single files from the ZIP also with only a little traffic.

Re: Peeking Inside Gigantic Zips with Only Kilobytes

#7

Post by aeblyve » Thu, Oct 16, 2025, 8:17 PM UTC

This is also quite easy to do with .tar files, not to be confused with .tar.gz files.

Re: Peeking Inside Gigantic Zips with Only Kilobytes

#8

Post by toomuchtodo » Thu, Oct 16, 2025, 8:18 PM UTC

gildas wrote: post #3

For implementation in a library, you can use HttpRangeReader [1][2] in zip.js [3] (disclaimer: I am the author). It's a solid feature that has been in the library for about 10 years. [1] https://gildas-lormeau.github.io/zip.js/api/classes/HttpRang... [2] https://github.com/gildas-lormeau/zip.js/blob/master/tests/a... [3] https://github.com/gildas-lormeau/zip.js

Based on your experience, is zip the optimal archive format for long term digital archival in object storage if the use case calls for reading archives via http for scanning and cherry picking? Or is there a more optimal archive format?

Re: Peeking Inside Gigantic Zips with Only Kilobytes

#9

Post by Lammy » Thu, Oct 16, 2025, 8:25 PM UTC

> That question took me into the guts of the ZIP format, where I learned there’s a tiny index at the end that points to everything else.

Tangential, but any Free Software that uses `shared-mime-info` to identify files (any of your GNOMEs, KDEs, etc) are unable to correctly identify Zip files by their EOCD due to lack of accepted syntax for defining search patterns based on negative file offsets. Please show your support on this Issue if you would also like to see this resolved: https://gitlab.freedesktop.org/xdg/shared-mime-info/-/issues... (linking to my own comment, so no this is not brigading)

Anything using `file(1)` does not have this problem: https://github.com/file/file/blob/280e121/magic/Magdir/zip#L...

Re: Peeking Inside Gigantic Zips with Only Kilobytes

#10

Post by xp84 » Thu, Oct 16, 2025, 8:42 PM UTC

xg15 wrote: post #4

This is really cool! Could also make a useful standalone command line tool. I think the general pattern - using the range header + prior knowledge of a file format to only download the parts of a file that are relevant - is still really underutilized. One small problem I see is that a server that does not support range requests would just try to send you the entire file in the first request, I think. So maybe doing a…

How common is it in practice today to not support ranges? I remember back in the early days of broadband (c. 2000) when having a Download Manager was something most nerds endorsed, that most servers then supported partial downloads. Aside from toy projects has anyone encountered a server which didn't allow ranges (unless specifically configured to forbid it)?