Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Full-history English Wikipedia dump produced: 5.6TB uncompressed, 32GB 7z'd

infodisiac.com

Post reply on HN
Print view View original

31–35 of 35 posts

Re: Full-history English Wikipedia dump produced: 5.6TB uncompressed, 32GB 7z'd

#31

Post by JoachimSchipper »

_delirium wrote: post #24
MikeCapone wrote: post #19

> 5.6 Tb uncompressed, 280 Gb in bz2 compression format, 32 Gb in 7z compression format Wow, I didn't know 7z was this much better than bz2. Is this the expected result, or is there something special with Wikipedia that plays to the strengths of 7z?

I'd guess it has to do mainly with 7z being able to use a larger block size, while bzip2's is 900kb; and possibly being able to do something better with large runs of repeated text. There are large articles with hundreds of revisions in a row that leave most of the content unchanged; [[George W. Bush]], for example, is around 180kb per revision, and is edited a lot , mostly with minor changes. So bzip2's block size m…

Does Wikipedia really store every single revision of every single file? As in, not deltas? Why is it done that way?

Re: Full-history English Wikipedia dump produced: 5.6TB uncompressed, 32GB 7z'd

#32

Post by _delirium »

JoachimSchipper wrote: post #31
_delirium wrote: post #24

Earlier quoted context omitted.

I'd guess it has to do mainly with 7z being able to use a larger block size, while bzip2's is 900kb; and possibly being able to do something better with large runs of repeated text. There are large articles with hundreds of revisions in a row that leave most of the content unchanged; [[George W. Bush]], for example, is around 180kb per revision, and is edited a lot , mostly with minor changes. So bzip2's block size m…

Does Wikipedia really store every single revision of every single file? As in, not deltas? Why is it done that way?

In the dump, I think for robustness and ease of extracting subsets.

Robustness: Having to essentially play back a log to recover any particular revision increases a chance of something eventually getting corrupted, and so it's somewhat safer to avoid it in something intended to be archival.

Ease of extracting subsets: For researchers, having the revisions be independent allows you to filter the XML dump through a SAX parser (or similar) to grab only revisions meeting particular criteria. If deltas were stored, you'd have to reconstruct those revisions from the deltas, which would make it really expensive to do things like, "I want to look at every article as it appeared at noon on April 1, 2007".

In the live DB, I think just because it's cheaper to get a ton of storage, esp. for rarely-retrieved old revisions, than to add the overhead of computing deltas and applying them to reconstruct revisions. In particular, you'd have to compute a diff for every edit in that situation, whereas currently MediaWiki only computes diffs when a user requests to view one from the "history" tab, which is a tiny proportion of all edits.

Re: Full-history English Wikipedia dump produced: 5.6TB uncompressed, 32GB 7z'd

#33

Post by res0nat0r »

_delirium wrote: post #24
MikeCapone wrote: post #19

> 5.6 Tb uncompressed, 280 Gb in bz2 compression format, 32 Gb in 7z compression format Wow, I didn't know 7z was this much better than bz2. Is this the expected result, or is there something special with Wikipedia that plays to the strengths of 7z?

I'd guess it has to do mainly with 7z being able to use a larger block size, while bzip2's is 900kb; and possibly being able to do something better with large runs of repeated text. There are large articles with hundreds of revisions in a row that leave most of the content unchanged; [[George W. Bush]], for example, is around 180kb per revision, and is edited a lot , mostly with minor changes. So bzip2's block size m…

It would be cool to see how the 7z archive compares to rzip, which (I think) might have the largest input size window of them all (up to 900MB). This software was written by Andrew Tridgell of rsync fame. It's major drawback is that it cant work on stdio and uses tons of RAM.

Re: Full-history English Wikipedia dump produced: 5.6TB uncompressed, 32GB 7z'd

#34

Post by ajb »

bprater wrote: post #27

One wonders if this will be the file first fed into something approximating machine consciousness. I'm not sure where else you can easily get such a high-quantity of fairly consistent human interest data. Quick question: what does "bot-edited" entries refer to?

bots are used for hugely common editing operations, such as various kinds on cleanup.

Re: Full-history English Wikipedia dump produced: 5.6TB uncompressed, 32GB 7z'd

#35

Post by JoachimSchipper »

_delirium wrote: post #32
JoachimSchipper wrote: post #31

Earlier quoted context omitted.

Does Wikipedia really store every single revision of every single file? As in, not deltas? Why is it done that way?

In the dump, I think for robustness and ease of extracting subsets. Robustness: Having to essentially play back a log to recover any particular revision increases a chance of something eventually getting corrupted, and so it's somewhat safer to avoid it in something intended to be archival. Ease of extracting subsets: For researchers, having the revisions be independent allows you to filter the XML dump through a SAX…

I understand it's simpler to store everything, and simplicity _is_ a virtue; but one could store the current revision plus deltas (and perhaps a few intermediate revisions for oft-edited articles), and obtain performance at least as good as in the current case. It would also save lots of space.
Post reply on HN