Earlier quoted context omitted.
That's because, despite what the 14 year olds on digg and reddit think, this isn't for you to download on your computer at your house. This is for archival or data-mining purposes. I apologize for the minor insult at digg/reddit, I just remember a few years ago a link to the archive was posted on digg and everyone started downloading it...unnecessarily wasting wikipedia's limited and donated resources.
If that's a problem, they could have put it up with bittorrent and throttled the bandwidth.
Full-history English Wikipedia dump produced: 5.6TB uncompressed, 32GB 7z'd
21–30 of 35 posts
Re: Full-history English Wikipedia dump produced: 5.6TB uncompressed, 32GB 7z'd
#22Doesn't include deleted articles, so no hope if you want to recover one of them. This is a pity since Wikipedia deletes too many articles.
Re: Full-history English Wikipedia dump produced: 5.6TB uncompressed, 32GB 7z'd
#23Impressive...I wonder how big a content snapshot is, ie no article histories and no meta-material like talk pages or WP:xxx pages, just the user-facing content. I was also sort of hoping to see from the stats what proportion of content was public-facing vs devoted to arguments between wikipedians...if you look at the stats for 'most edited articles' (accessible from the top link) it's interesting that of the top 50 m…
> I wonder how big a content snapshot is, ie no article histories and no meta-material like talk pages or WP:xxx pages, just the user-facing content I don't know how big it is uncompressed, but they do have a dump of just that part: 2010-03-16 08:44:40 done Articles, templates, image descriptions, and primary meta-pages. 2010-03-16 08:44:40: enwiki 9654328 pages (255.402/sec), 9654328 revs (255.402/sec), 82.9% prefet…
Re: Full-history English Wikipedia dump produced: 5.6TB uncompressed, 32GB 7z'd
#24> 5.6 Tb uncompressed, 280 Gb in bz2 compression format, 32 Gb in 7z compression format Wow, I didn't know 7z was this much better than bz2. Is this the expected result, or is there something special with Wikipedia that plays to the strengths of 7z?
IIRC from some tests a year or so ago, Wikipedia hasn't found any significant improvement from 7z over bz2 on the current-revisions-only dump, which looks more like just normal English text; that's why it doesn't bother to provide a separate 7z version of that. It seems to only be this pattern of [200kb article][almost the same 200kb article][almost the same 200kb article again] that 7z kills bz2 on.
Re: Full-history English Wikipedia dump produced: 5.6TB uncompressed, 32GB 7z'd
#25Doesn't include deleted articles, so no hope if you want to recover one of them. This is a pity since Wikipedia deletes too many articles.
That's talked about on and off, but one issue is that they'd have to filter deleted articles by deletion reason, at least broadly into "deleted for legal reasons" and "deleted for non-legal reasons" bins. There'd be no problem distributing a dump of articles deleted due to non-notability, but a dump of articles axed for copyright violation, libel, or other legal issues would be a problem. For specific articles delete…
Wrong forum for these suggestions but I've never had the time or inclination to attempt to reach the Wikipedia inner sanctum.
Re: Full-history English Wikipedia dump produced: 5.6TB uncompressed, 32GB 7z'd
#26Earlier quoted context omitted.
> I wonder how big a content snapshot is, ie no article histories and no meta-material like talk pages or WP:xxx pages, just the user-facing content I don't know how big it is uncompressed, but they do have a dump of just that part: 2010-03-16 08:44:40 done Articles, templates, image descriptions, and primary meta-pages. 2010-03-16 08:44:40: enwiki 9654328 pages (255.402/sec), 9654328 revs (255.402/sec), 82.9% prefet…
Well spotted. This has great possibilities for education in the 3rd world.
Re: Full-history English Wikipedia dump produced: 5.6TB uncompressed, 32GB 7z'd
#27Quick question: what does "bot-edited" entries refer to?
Re: Full-history English Wikipedia dump produced: 5.6TB uncompressed, 32GB 7z'd
#28Earlier quoted context omitted.
That's talked about on and off, but one issue is that they'd have to filter deleted articles by deletion reason, at least broadly into "deleted for legal reasons" and "deleted for non-legal reasons" bins. There'd be no problem distributing a dump of articles deleted due to non-notability, but a dump of articles axed for copyright violation, libel, or other legal issues would be a problem. For specific articles delete…
Surely articles have posted on them, or their talk pages, the reason for deletion to allow for responses? Also there should be some way for people to find that an article has been deleted so that they don't recreate the article and repeat the error. Indeed rather than delete couldn't a placeholder be implemented. Wrong forum for these suggestions but I've never had the time or inclination to attempt to reach the Wiki…
You do get a bit of a warning if you recreate a deleted page. When you go to the editing screen at the title of an article that was previously deleted, it'll show you the summary from the deletion log at the top, and ask you if you're sure you want to recreate it. There's also a "nothing can go here" protected placeholder used for articles that are persistently being recreated, which'll make it impossible to edit at that location.
Yeah, I can sympathize on the Wikipedia-inner-sanctum thing. I was actually pretty deeply into it (I've been an admin since '04, was formerly on the Arbitration Committee, formerly active on the mailing lists, etc.), but as the Policy And Process kept accumulating, I lost interest in navigating it, so am more on the periphery these days. It's probably inevitable that things would go that direction, because in the early days there were probably <100 Wikipedians active enough to form the Wikipedia Cabal, all of whom at least recognized each others' names, so stuff could be pretty informal. But it's hard to scale that up to a site with 1700 admins and 15k+ editors. A lot of things are kind of lame about how things are organized these days, but honestly I have no idea how I'd do it better; despite its flaws it's often still amazing to me that Wikipedia works at all.
Re: Full-history English Wikipedia dump produced: 5.6TB uncompressed, 32GB 7z'd
#29Earlier quoted context omitted.
7zip, like many other compression schemes, is optimized and designed so that decompression is typically (much) faster than compression. The web page ( http://www.7-zip.org/7z.html ) states that the default "native" LZMA format decompresses at between 10 and 20 times the speed that it compresses. So, 15 days / 15 is about one day to decompress, then.
LZMA is well-known for its decompression speed. This is one of the reasons it's a popular choice for filesystem compression. It's quite easy for LZMA to keep a pretty fair pace with a disk, so you get a pretty noticeable performance boost by adding LZMA at the filesystem layer, especially for read-heavy workloads. Gzip usually gets a slightly better compression ratio, but at the expense of decompression speed, partic…
At least it does when I test it. But it's slower - sometimes much slower (depends on settings).
Re: Full-history English Wikipedia dump produced: 5.6TB uncompressed, 32GB 7z'd
#30Earlier quoted context omitted.
LZMA is well-known for its decompression speed. This is one of the reasons it's a popular choice for filesystem compression. It's quite easy for LZMA to keep a pretty fair pace with a disk, so you get a pretty noticeable performance boost by adding LZMA at the filesystem layer, especially for read-heavy workloads. Gzip usually gets a slightly better compression ratio, but at the expense of decompression speed, partic…
I'm pretty sure LZMA (7z) compresses better than bz2. At least it does when I test it. But it's slower - sometimes much slower (depends on settings).
Turns out the paper I was recalling and referencing dealt with LZO, not LZMA, so maybe I have less to say about LZMA than I thought. Shows how much you jerks read before upvoting. ;-)