Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Full-history English Wikipedia dump produced: 5.6TB uncompressed, 32GB 7z'd

infodisiac.com

Post reply on HN
Print view View original

11–20 of 35 posts

Re: Full-history English Wikipedia dump produced: 5.6TB uncompressed, 32GB 7z'd

#11

Post by [deleted] »

blhack wrote: post #9
kez wrote: post #4

Interesting, but somehow doubt that many people have the set-up to handle this number of data.

That's because, despite what the 14 year olds on digg and reddit think, this isn't for you to download on your computer at your house. This is for archival or data-mining purposes. I apologize for the minor insult at digg/reddit, I just remember a few years ago a link to the archive was posted on digg and everyone started downloading it...unnecessarily wasting wikipedia's limited and donated resources.

[deleted]

Re: Full-history English Wikipedia dump produced: 5.6TB uncompressed, 32GB 7z'd

#12

Post by eru »

blhack wrote: post #9
kez wrote: post #4

Interesting, but somehow doubt that many people have the set-up to handle this number of data.

That's because, despite what the 14 year olds on digg and reddit think, this isn't for you to download on your computer at your house. This is for archival or data-mining purposes. I apologize for the minor insult at digg/reddit, I just remember a few years ago a link to the archive was posted on digg and everyone started downloading it...unnecessarily wasting wikipedia's limited and donated resources.

If that's a problem, they could have put it up with bittorrent and throttled the bandwidth.

Re: Full-history English Wikipedia dump produced: 5.6TB uncompressed, 32GB 7z'd

#13

Post by mukyu »

unwind wrote: post #10
helwr wrote: post #7

40 + 15 days to compress? how long it would take to decompress this thing

7zip, like many other compression schemes, is optimized and designed so that decompression is typically (much) faster than compression. The web page ( http://www.7-zip.org/7z.html ) states that the default "native" LZMA format decompresses at between 10 and 20 times the speed that it compresses. So, 15 days / 15 is about one day to decompress, then.

They also parallelize the compression (at least with bzip2) and run more than one dump process at the same time.

Re: Full-history English Wikipedia dump produced: 5.6TB uncompressed, 32GB 7z'd

#14

Post by _delirium »

philwelch wrote: post #5

Doesn't include deleted articles, so no hope if you want to recover one of them. This is a pity since Wikipedia deletes too many articles.

That's talked about on and off, but one issue is that they'd have to filter deleted articles by deletion reason, at least broadly into "deleted for legal reasons" and "deleted for non-legal reasons" bins. There'd be no problem distributing a dump of articles deleted due to non-notability, but a dump of articles axed for copyright violation, libel, or other legal issues would be a problem.

For specific articles deleted for non-legal reasons (most commonly notability), you can get a copy from a WP admin. Some have volunteered themselves as willing to answer requests: http://en.wikipedia.org/wiki/Category:Wikipedia_administrato...

Re: Full-history English Wikipedia dump produced: 5.6TB uncompressed, 32GB 7z'd

#15

Post by _delirium »

blhack wrote: post #8
helwr wrote: post #7

40 + 15 days to compress? how long it would take to decompress this thing

I imagine that the compression is only for saving time during transfer. That, and hopefully you could decompress just what you needed. I.E. a tarball of compressed articles.

A lot of researchers just stream the decompressed stream directly from 7z into their analysis scripts. If you were to actually decompress to a giant XML file first, you'd both: 1. need a 6-TB drive; and 2. start getting disk I/O as a big bottleneck. In a lot of cases, the analysis scripts, rather than decompression, are the bottleneck anyway: 7z can feed you data faster than your XML parser and scripts can consume it.

Re: Full-history English Wikipedia dump produced: 5.6TB uncompressed, 32GB 7z'd

#16

Post by _delirium »

jonknee wrote: post #6

The article says Tb not TB, but in reality it appears to be TB. That's quite a difference. Still seems heavy for text, but I assume the full text of every revision is in it, not just diffs.

Yeah, every revision is standalone, which is why it compresses so well (obviously there are a lot of edits that make relatively small changes). One reason is to make it easier for researchers to grab specific revisions, e.g. run the dump through a filter returning only revisions as of June 1, 2006--- without having to apply a ton of diffs to reconstruct those revisions.

The dump schema is something like:

  
    
      some metadata
    
    
      Article Title
      15580374
      
        139992
        2002-01-26T15:28:12Z
        
          _delirium
          82
        
        vandalized this page
        Complete text of this revision of the article goes here.
        
      
      
        ...next revision of this page...
      
    
    
      ...revisions of the next page...

Re: Full-history English Wikipedia dump produced: 5.6TB uncompressed, 32GB 7z'd

#17

Post by leif »

unwind wrote: post #10
helwr wrote: post #7

40 + 15 days to compress? how long it would take to decompress this thing

7zip, like many other compression schemes, is optimized and designed so that decompression is typically (much) faster than compression. The web page ( http://www.7-zip.org/7z.html ) states that the default "native" LZMA format decompresses at between 10 and 20 times the speed that it compresses. So, 15 days / 15 is about one day to decompress, then.

LZMA is well-known for its decompression speed. This is one of the reasons it's a popular choice for filesystem compression. It's quite easy for LZMA to keep a pretty fair pace with a disk, so you get a pretty noticeable performance boost by adding LZMA at the filesystem layer, especially for read-heavy workloads.

Gzip usually gets a slightly better compression ratio, but at the expense of decompression speed, particularly on less compressible data (LZMA somehow seems to know better when to give up trying to compress). Bzip2 has the best compression ratio of the three, but is far too slow to compress and decompress, so you end up losing more time decompressing than you gained by doing less actual I/O.

EDIT: source, for those curious folk out there: http://portal.acm.org/citation.cfm?id=1534536 (caveat: the experiments were run by taking a large file on disk and compressing it to another large file on disk, so seek thrashing may have been an issue and I'm not quick to take the numbers for all they should be worth)

Re: Full-history English Wikipedia dump produced: 5.6TB uncompressed, 32GB 7z'd

#18

Post by anigbrowl »

Impressive...I wonder how big a content snapshot is, ie no article histories and no meta-material like talk pages or WP:xxx pages, just the user-facing content.

I was also sort of hoping to see from the stats what proportion of content was public-facing vs devoted to arguments between wikipedians...if you look at the stats for 'most edited articles' (accessible from the top link) it's interesting that of the top 50 most edited articles, only one, 'George W. Bush' is user-facing - and I suspect that only made it in because of persistent vandalism.

Still, with history and all included, there is some fabulous data-mining potential here, with which there's the potential to do some really innovative work. I'd hazard a guess that the size of Wikipedia already exceeds that of existing language corpuses like the US code...

/retreats into corner muttering about semantic engines and link free concepts of total hypertext as necessary AI boot conditions

Re: Full-history English Wikipedia dump produced: 5.6TB uncompressed, 32GB 7z'd

#19

Post by MikeCapone »

> 5.6 Tb uncompressed, 280 Gb in bz2 compression format, 32 Gb in 7z compression format

Wow, I didn't know 7z was this much better than bz2. Is this the expected result, or is there something special with Wikipedia that plays to the strengths of 7z?

Re: Full-history English Wikipedia dump produced: 5.6TB uncompressed, 32GB 7z'd

#20

Post by _delirium »

anigbrowl wrote: post #18

Impressive...I wonder how big a content snapshot is, ie no article histories and no meta-material like talk pages or WP:xxx pages, just the user-facing content. I was also sort of hoping to see from the stats what proportion of content was public-facing vs devoted to arguments between wikipedians...if you look at the stats for 'most edited articles' (accessible from the top link) it's interesting that of the top 50 m…

> I wonder how big a content snapshot is, ie no article histories and no meta-material like talk pages or WP:xxx pages, just the user-facing content

I don't know how big it is uncompressed, but they do have a dump of just that part:

  2010-03-16 08:44:40 done Articles, templates, image descriptions, and primary meta-pages.
  2010-03-16 08:44:40: enwiki 9654328 pages (255.402/sec), 9654328 revs (255.402/sec), 82.9% prefetched, ETA 2010-03-17 03:08:26 [max 26568677]
  This contains current versions of article content, and is the archive most mirror sites will probably want.
  pages-articles.xml.bz2 5.7 GB
Post reply on HN