Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Wikipedia: Database Download

en.wikipedia.org

Post reply on HN
Print view View original

1–10 of 106 posts

Re: Wikipedia: Database Download

#4

Post by _fat_santa »

cynicalsecurity wrote: post #3

Okay, that looked a bit ridiculous in the pre-AI era (who needs to download the whole Wikipedia?), but now I can see the sense in it.

Oh no even now there is plenty of use for it outside of AI training. Just think of all the schools in villages all around the world that don't have access to the internet or have a very limited connection. I've worked with folks that would setup local "wikipedia servers" for schools so that kids could access Wikipedia via a local network connection. In other setups they just download all of wikipedia to a set of laptops and you use one of the offline readers to browse it.

This is essentially the modern version of having a library of encyclopedias.

Re: Wikipedia: Database Download

#5

Post by jsheard »

cynicalsecurity wrote: post #3

Okay, that looked a bit ridiculous in the pre-AI era (who needs to download the whole Wikipedia?), but now I can see the sense in it.

Too bad the AI scrapers don't care, and are melting Wikipedia's production servers anyway.

https://arstechnica.com/information-technology/2025/04/ai-bo...

Re: Wikipedia: Database Download

#7

Post by sdoering »

jsheard wrote: post #5
cynicalsecurity wrote: post #3

Okay, that looked a bit ridiculous in the pre-AI era (who needs to download the whole Wikipedia?), but now I can see the sense in it.

Too bad the AI scrapers don't care, and are melting Wikipedia's production servers anyway. https://arstechnica.com/information-technology/2025/04/ai-bo...

Tragedy of the commons. And that’s why we can’t have nice things.

Because people are people. And will always prioritize egotism over respect for the common good.

Re: Wikipedia: Database Download

#8

Post by ozmodiar »

_fat_santa wrote: post #4
cynicalsecurity wrote: post #3

Okay, that looked a bit ridiculous in the pre-AI era (who needs to download the whole Wikipedia?), but now I can see the sense in it.

Oh no even now there is plenty of use for it outside of AI training. Just think of all the schools in villages all around the world that don't have access to the internet or have a very limited connection. I've worked with folks that would setup local "wikipedia servers" for schools so that kids could access Wikipedia via a local network connection. In other setups they just download all of wikipedia to a set of lapt…

I'm thinking less about AI training and more about having a source of (reasonably) reliable information from the net, in case AI generated fake images and generated cross referenced texts start making it too difficult to discern real history from malicious rewrites. It's bad enough now, but can get much worse with the proliferation of AI agents.

Re: Wikipedia: Database Download

#9

Post by hagbard_c »

cynicalsecurity wrote: post #3

Okay, that looked a bit ridiculous in the pre-AI era (who needs to download the whole Wikipedia?), but now I can see the sense in it.

> who needs to download the whole Wikipedia

Anyone who wants to have access while off-line, for whatever reason. This can be as simple as saving costs via more complicated as accessing content from regions with spotty and/or expensive connectivity (you're on a ship out of reach of shore-based mobile networks, you do not have access to Starlink or something similar, you're deep in the jungle, deep underground, etc) to some prepper scenario where connectivity ends at the cave entry because the 'net has ceased to exist.

I would like to have a less politically biased online encyclopedia for the latter scenario, it would be a shame to start a new society based on the same bad ideas which brought down the previous one. If ever a politically neutral LLM becomes available that'd be one of the first tasks I'd put it to: point out bias - any bias - in articles, encyclopedias and other 'sources' (yes, I know, WP is not an original source but for this purpose it is) of knowledge.

Post reply on HN