Live data from Hacker News

Wikipedia: Database Download

en.wikipedia.org

71–80 of 106 posts

Re: Wikipedia: Database Download

#71
post #62

You can also get it as a .zim file for easy offline browsing with Kiwix. The whole enchilada: https://download.kiwix.org/zim/wikipedia/wikipedia_en_all_ma... Other versions: https://library.kiwix.org/#lang=eng&category=wikipedia

Will the 2025 zim be available as well?

I am wondering the same thing. I have Jan 2021, Jan 2024... I want to keep a snapshot each year and I wonder why a new one hasn't been generated.

I haven't looked for documentation on creating my own zim file.

Re: Wikipedia: Database Download

#73
post #37

Earlier quoted context omitted.

I bet someone like Cloudflare could pull the dataset each day and serve up a plain text/Markdown version of Wikipedia for rounding error levels of spend. I just loaded a random Wikipedia page and it had a weight of 1.5MB in all for what I worked out would be about 30KB of Markdown (i.e. 50x less bandwidth). Of course, the problem then is getting all these scrapers and bots to actually use the alternative, but Wikimed…

Someone suggested to me to apply a filter that serves .md or txt to bots/ai scrapers instead of the regular website, seems smart if it works but i hate it when i get captchas and this could end up similarly detecting non-bots as bots maybe a view full website link loaded on js so bots dont see it idk

I would love to see most sites serve me markdown. I'd happily install a browser extension to mask me as a a AI bot scraper if it means I can just get the text without all the noise.

Re: Wikipedia: Database Download

#74

I wonder how easy it would be to make a practically indestructible, everlasting Wikipedia reader. Something using solar for power, with a rugged and water-resistant enclosure, made of extremely high-quality components that won't break for hundreds of years at least. Maybe add an IRDA port for good measure, to make it possible to transfer all the data out somewhat quickly. You could make hundreds of these and put them…

You could even make it radiation tolerant by printing it.

Relevant XKCD: https://what-if.xkcd.com/59/

Re: Wikipedia: Database Download

#75
post #42

If you've got some spare bandwidth & storage then seeding some of the torrents here is a cheap and fun way of helping Wikipedia out. I've served around 20TB of these dumps in the past year. https://meta.wikimedia.org/wiki/Data_dump_torrents

Do you happen to know why wikipedia didn't embrace torrents as the default download method?

Most people don't use torrents.

Re: Wikipedia: Database Download

#76

I wonder how easy it would be to make a practically indestructible, everlasting Wikipedia reader. Something using solar for power, with a rugged and water-resistant enclosure, made of extremely high-quality components that won't break for hundreds of years at least. Maybe add an IRDA port for good measure, to make it possible to transfer all the data out somewhat quickly. You could make hundreds of these and put them…

Kiwix has created pretty polished software for this: https://kiwix.org/

My last download of English Wikipedia was ~110 GB and includes images! It's impressively small for the volume of information available.

Re: Wikipedia: Database Download

#78

I wonder how easy it would be to make a practically indestructible, everlasting Wikipedia reader. Something using solar for power, with a rugged and water-resistant enclosure, made of extremely high-quality components that won't break for hundreds of years at least. Maybe add an IRDA port for good measure, to make it possible to transfer all the data out somewhat quickly. You could make hundreds of these and put them…

Aard2 for Android exists since at least 2015:

https://f-droid.org/packages/itkach.aard2

I have many current and old dumps and can switch between a few years. Very nice in case of deleted articles or to check old time stamped versions. It also supports more than just Wikipedia like wikiquote or wikivoyage or cooking wiki. You can compile own mediawikis too

Re: Wikipedia: Database Download

#79
post #21

You can also get it as a .zim file for easy offline browsing with Kiwix. The whole enchilada: https://download.kiwix.org/zim/wikipedia/wikipedia_en_all_ma... Other versions: https://library.kiwix.org/#lang=eng&category=wikipedia

I tried this kiwix the other day, it has like a 300mb "essentials" text version that was interesting. This comment was downvoted and instead, it'd better merit a comment as to "why" it wasn't contributing to the discussion?

> I tried this kiwix the other day, it has like a 300mb "essentials" text version that was interesting.

I didn't downvote the comment, but it's not an incredibly deep contribution, is it?

If you really wish to contribute, perhaps you can say what the "'essentials' text version" contained and why you found it interesting?

Re: Wikipedia: Database Download

#80
post #50

Earlier quoted context omitted.

It's not Wikipedia's fault that the vast majority of right wing media consists of pure propaganda, disinformation, and lies.

[citation needed] And you know, I wouldn't be surprised if people hurling those accusations somehow believe that the lies and misinformation are one-sided and partisan. As if leftism has some sort of monopoly on Truth and Goodness bestowed from above. It's really been sickening to see the media outlets just lay down thick trails of bullshit that is designed to distract us, to instill fear, uncertainty, and doubt, to…

Wait why just leftism, what happened to centrism ? Where'd the goalposts go ?
Post reply on HN