Full-history English Wikipedia dump produced: 5.6TB uncompressed, 32GB 7z'd

#1

Post by chl » Wed, Apr 14, 2010, 3:28 PM UTC

Full-history English Wikipedia dump produced: 5.6TB uncompressed, 32GB 7z'd
infodisiac.com

Re: Full-history English Wikipedia dump produced: 5.6TB uncompressed, 32GB 7z'd

#2

Post by baddox » Wed, Apr 14, 2010, 3:44 PM UTC

I like the quick fix the site designer used to switch from a static layout to a fluid one.

Re: Full-history English Wikipedia dump produced: 5.6TB uncompressed, 32GB 7z'd

#3

Post by [deleted] » Wed, Apr 14, 2010, 3:47 PM UTC

[deleted]

Re: Full-history English Wikipedia dump produced: 5.6TB uncompressed, 32GB 7z'd

#4

Post by kez » Wed, Apr 14, 2010, 3:47 PM UTC

Interesting, but somehow doubt that many people have the set-up to handle this number of data.

Re: Full-history English Wikipedia dump produced: 5.6TB uncompressed, 32GB 7z'd

#5

Post by philwelch » Wed, Apr 14, 2010, 3:50 PM UTC

Doesn't include deleted articles, so no hope if you want to recover one of them. This is a pity since Wikipedia deletes too many articles.

Re: Full-history English Wikipedia dump produced: 5.6TB uncompressed, 32GB 7z'd

#6

Post by jonknee » Wed, Apr 14, 2010, 3:53 PM UTC

The article says Tb not TB, but in reality it appears to be TB. That's quite a difference. Still seems heavy for text, but I assume the full text of every revision is in it, not just diffs.

Re: Full-history English Wikipedia dump produced: 5.6TB uncompressed, 32GB 7z'd

#7

Post by helwr » Wed, Apr 14, 2010, 4:03 PM UTC

40 + 15 days to compress? how long it would take to decompress this thing

Re: Full-history English Wikipedia dump produced: 5.6TB uncompressed, 32GB 7z'd

#8

Post by blhack » Wed, Apr 14, 2010, 4:06 PM UTC

helwr wrote: post #7

40 + 15 days to compress? how long it would take to decompress this thing

I imagine that the compression is only for saving time during transfer.

That, and hopefully you could decompress just what you needed. I.E. a tarball of compressed articles.

Re: Full-history English Wikipedia dump produced: 5.6TB uncompressed, 32GB 7z'd

#9

Post by blhack » Wed, Apr 14, 2010, 4:23 PM UTC

kez wrote: post #4

Interesting, but somehow doubt that many people have the set-up to handle this number of data.

That's because, despite what the 14 year olds on digg and reddit think, this isn't for you to download on your computer at your house. This is for archival or data-mining purposes.

I apologize for the minor insult at digg/reddit, I just remember a few years ago a link to the archive was posted on digg and everyone started downloading it...unnecessarily wasting wikipedia's limited and donated resources.

Re: Full-history English Wikipedia dump produced: 5.6TB uncompressed, 32GB 7z'd

#10

Post by unwind » Wed, Apr 14, 2010, 4:27 PM UTC

helwr wrote: post #7

40 + 15 days to compress? how long it would take to decompress this thing

7zip, like many other compression schemes, is optimized and designed so that decompression is typically (much) faster than compression.

The web page (http://www.7-zip.org/7z.html) states that the default "native" LZMA format decompresses at between 10 and 20 times the speed that it compresses.

So, 15 days / 15 is about one day to decompress, then.