Live data from Hacker News

Wikipedia: Database Download

en.wikipedia.org

21–30 of 106 posts

Re: Wikipedia: Database Download

#21

You can also get it as a .zim file for easy offline browsing with Kiwix. The whole enchilada: https://download.kiwix.org/zim/wikipedia/wikipedia_en_all_ma... Other versions: https://library.kiwix.org/#lang=eng&category=wikipedia

I tried this kiwix the other day, it has like a 300mb "essentials" text version that was interesting.

This comment was downvoted and instead, it'd better merit a comment as to "why" it wasn't contributing to the discussion?

Re: Wikipedia: Database Download

#23
post #6

Okay, that looked a bit ridiculous in the pre-AI era (who needs to download the whole Wikipedia?), but now I can see the sense in it.

Also helps save Wikipedia if it gets shut down - which might happen!

True. Musk for example is publicly attacking it for spreading "left-wing lies" because in his wiki page there are statements like "He has been criticized for making unscientific and misleading statements, including COVID-19 misinformation and promoting conspiracy theories, and affirming antisemitic, racist, and transphobic comments." which are just pure facts.

It would be nice to have something like this more decentralized.

Re: Wikipedia: Database Download

#25
post #18

Might also be worthwhile to download a pre ~2023 dump, because Low-background steel.

just in case anyone is as obtuse as I am, I believe the joke here is that the contents of Wikipedia might be contaminated with AI generated content starting around 2023. you can probably look up low background steel to complete the analogy.

Re: Wikipedia: Database Download

#26

Okay, that looked a bit ridiculous in the pre-AI era (who needs to download the whole Wikipedia?), but now I can see the sense in it.

Oh no even now there is plenty of use for it outside of AI training. Just think of all the schools in villages all around the world that don't have access to the internet or have a very limited connection. I've worked with folks that would setup local "wikipedia servers" for schools so that kids could access Wikipedia via a local network connection. In other setups they just download all of wikipedia to a set of lapt…

There's already a project to serve the use case you're describing (school in a disconnected village): Internet in a Box

https://internet-in-a-box.org/

They provide offline access to Wikipedia, OpenStreetMap, Project Gutenberg, and many other resources.

Re: Wikipedia: Database Download

#28
post #8

Earlier quoted context omitted.

I'm thinking less about AI training and more about having a source of (reasonably) reliable information from the net, in case AI generated fake images and generated cross referenced texts start making it too difficult to discern real history from malicious rewrites. It's bad enough now, but can get much worse with the proliferation of AI agents.

Pre-2022 Wikipedia dumps will be analyzed by future historians.

And banned by future governments.

Re: Wikipedia: Database Download

#29

Okay, that looked a bit ridiculous in the pre-AI era (who needs to download the whole Wikipedia?), but now I can see the sense in it.

No idea when your pre-AI era begun, but I was much more excited to host Wikipedia locally 15 years ago than I am now.

Re: Wikipedia: Database Download

#30
post #18

Might also be worthwhile to download a pre ~2023 dump, because Low-background steel.

> download a pre ~2023 dump, because Low-background steel.

Military A.I. was likely in use earlier, and since PSYOPS are the most used and most effective weapon in the U.S. Military's arsenal, you absolutely know it was used. It ain't a war crime the first time...

Post reply on HN