Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Wikipedia: Database Download

en.wikipedia.org

Post reply on HN
Print view View original

31–40 of 106 posts

Re: Wikipedia: Database Download

#31

Post by ujkhsjkdhf234 »

jsheard wrote: post #5
cynicalsecurity wrote: post #3

Okay, that looked a bit ridiculous in the pre-AI era (who needs to download the whole Wikipedia?), but now I can see the sense in it.

Too bad the AI scrapers don't care, and are melting Wikipedia's production servers anyway. https://arstechnica.com/information-technology/2025/04/ai-bo...

I would like companies to start aggressively pushing back against AI scrapers using things like Anubis[0]. If you can't be a good steward of the internet or respectful to other peoples' resources, then people have the right to deny them to you.

[0] https://github.com/TecharoHQ/anubis

Re: Wikipedia: Database Download

#36

Post by amanaplanacanal »

moritzwarhier wrote: post #13
hagbard_c wrote: post #9

Earlier quoted context omitted.

> who needs to download the whole Wikipedia Anyone who wants to have access while off-line, for whatever reason. This can be as simple as saving costs via more complicated as accessing content from regions with spotty and/or expensive connectivity (you're on a ship out of reach of shore-based mobile networks, you do not have access to Starlink or something similar, you're deep in the jungle, deep underground, etc) to…

Is there a "politically neutral" human? And if there was, what could that person reasonably say about politics?

I suspect "politically neutral" is a meaningless phrase. It's just a way for people to tar their political opponents by inference.

The problem is: even if you report only facts, there is an editorial function in choosing which facts to report, because it is physically impossible to report all facts. So someone can always point to some sort of bias on choosing which facts to report.

Re: Wikipedia: Database Download

#37

Post by tough »

petercooper wrote: post #20
jsheard wrote: post #5

Earlier quoted context omitted.

Too bad the AI scrapers don't care, and are melting Wikipedia's production servers anyway. https://arstechnica.com/information-technology/2025/04/ai-bo...

I bet someone like Cloudflare could pull the dataset each day and serve up a plain text/Markdown version of Wikipedia for rounding error levels of spend. I just loaded a random Wikipedia page and it had a weight of 1.5MB in all for what I worked out would be about 30KB of Markdown (i.e. 50x less bandwidth). Of course, the problem then is getting all these scrapers and bots to actually use the alternative, but Wikimed…

Someone suggested to me to apply a filter that serves .md or txt to bots/ai scrapers instead of the regular website, seems smart if it works but i hate it when i get captchas and this could end up similarly detecting non-bots as bots

maybe a view full website link loaded on js so bots dont see it idk

Re: Wikipedia: Database Download

#38

Post by froh »

icepat wrote: post #34
foreigner wrote: post #22

Earlier quoted context omitted.

LOL that is an amazing analogy, thank you.

My go-to as well. Pre-war steel.

it's not about pre-war. it's about pre-trinity-nuclear tests. which means uncontaminated by atmospheric radioactive isotopes. it happened at the end of ww-ii but that is not the point.

Re: Wikipedia: Database Download

#40

Post by tough »

amanaplanacanal wrote: post #36
moritzwarhier wrote: post #13

Earlier quoted context omitted.

Is there a "politically neutral" human? And if there was, what could that person reasonably say about politics?

I suspect "politically neutral" is a meaningless phrase. It's just a way for people to tar their political opponents by inference. The problem is: even if you report only facts, there is an editorial function in choosing which facts to report, because it is physically impossible to report all facts. So someone can always point to some sort of bias on choosing which facts to report.

And when editors have big ad spenders, you bet they won't criticize the hand that feeds them, most of the time
Post reply on HN