Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Wikipedia is struggling with voracious AI bot crawlers

engadget.com

Post reply on HN
Print view View original

81–90 of 105 posts

Re: Wikipedia is struggling with voracious AI bot crawlers

#81

Post by PeterStuer »

microtherion wrote: post #14

Not just Wikipedia. My home server (hosting a number of not particularly noteworthy things, such as my personal gitea instance) has been absolutely hammered in recent months, to the extent of periodically bringing down the server for hours with thrashing. The worst part is that every single sociopathic company in the world seems to have simultaneously unleashed their own fleet of crawlers. Most of the bots downright…

Why not just rate limit? IP range based blocking will likely hit far more legitimate users than you think.

Re: Wikipedia is struggling with voracious AI bot crawlers

#82

Post by marginalia_nu »

jerf wrote: post #42
roenxi wrote: post #22

Earlier quoted context omitted.

I've written some unfathomably bad web crawlers in the past. Indeed, web crawlers might be the most natural magnet for bad coding and eye-twitchingly questionable architectural practices I know of. While it likely isn't the major factor here I can attest that there are coders who see pages-articles-multistream.xml.bz2 and then reach for a wget + HTML parser combo. If you don't live and breath Wikipedia it is going to…

To be a "good" web crawler, you have to go beyond "not bad coding". If you just write the natural "fetch page, fetch next page, retry if it fails" loop, notably, missing any sort of wait between fetches, so that you fetch as quickly as possible, you are already a pest. You don't even need multiple threads or machines to be a pest; a single machine on a home connection fetching pages as quickly as it can be already be…

I think most crawlers inevitably tend to turn into spaghetti code because of the number of weird corner cases you need to deal with.

Crawlers are also incredibly difficult to test in a comprehensive way. No matter what test scenarios you come up with, there's a hundred more weird cases in the wild. (e.g. there's a world's difference between a server taking a long time to respond to a request, and a server sending headers quickly but taking a long time to send the body)

Re: Wikipedia is struggling with voracious AI bot crawlers

#84

Post by laz »

10 years ago at Facebook we had a systems design interview question called "botnet crawl" where the set up that I'd give would be:

I'm an entrepreneur who is going to get rich selling printed copies of Wikipedia. I'll pay you to fetch the content for me to print. You get 1000 compromised machines to use. Crawl Wikipedia and give me the data. Go.

Some candidates would (rightfully) point out that the entirety is available as an archive, so for "interviewing purposes" we'd have to ignore that fact.

If it went well, you would pivot back and forth: OK, you wrote a distributed crawler. Wikipedia hires you you to block it. What do you do? This cat and mouse game goes on indefinitely.

Re: Wikipedia is struggling with voracious AI bot crawlers

#85

Post by PeterStuer »

m101 wrote: post #31

Wikipedia spends 1% of its budget on hosting fees. It can spend a bit more given the rest of their corruptions.

Steady on cowboy. They might mislead the public in their needs and spends, but to jump from that to "corruption" is a bit too fast and loose.

From their financial statement 2024 you can learn that they probably spend about $6,825,794 on site operation (excl. salaries etc.). This includes $3,116,445 for Internet hosting and an estimated $3,709,349 on Server infrastructure (est. as 85% of equipment).

Now as of June 30, 2024, the Wikimedia Foundation's net assets totaled approximately $271.6 million, and the Wikimedia Endowment, established to support the long-term sustainability of Wikimedia projects, reported net assets of approximately $144.3 million as of the same date, so combined approximately $415.9 million.

So yes, annual sites operation is about 1,64% of their total assets, and they can operate all the wikimedia sites till the end of time without raising a single dime in donations ever again.

Sure they're not going to advertise this fact when doing another donation drive as that would likely make the donators starting to ask pertinent questions about the exact purpose of their donations, but that is just marketing, not "corruption".

Re: Wikipedia is struggling with voracious AI bot crawlers

#86

Post by PeterStuer »

insane_dreamer wrote: post #78
m101 wrote: post #31

Wikipedia spends 1% of its budget on hosting fees. It can spend a bit more given the rest of their corruptions.

> the rest of their corruptions the "corruption" accusations are mostly BS and the usual ideological differences I'll take Wikipedia, with all its warts, over $BigTech and $VC-driven (==Ad-driven) companies/orgs any day, and it's not even close.

You don't need a VC if your sitting on $415,9 Million.

Re: Wikipedia is struggling with voracious AI bot crawlers

#87

Post by jerf »

zokier wrote: post #71
jerf wrote: post #47

Earlier quoted context omitted.

Any plan that starts with "Step one: Apply the tool that almost perfectly distinguishes human traffic from non-human traffic" is doomed to failure. That's whatever the engineering equivalent of "begging the question" is, where the solution to the problem is that we assume that we have the solution to the problem.

Identity verification is not that far fetched these days. For europeans you got eIDAS and related tech, some other places have similar stuff, for rest of world you can do video based id checks. There are plenty of providers that handle this, it's pretty commonplace stuff.

That does not generically 100% solve the problem of "is this person a human". That ties things to an identity, but the verification that an identity is actually that human is not solved. Stolen identities, forged identities, faked identities, all still problems, and as soon as the full force of the black market capitalist market in such things is turned on that world, it'll still be a big problem.

Also video-based ID checks have a shelf-life measured in single-digit years now, if indeed the plural is even appropriate. The tricks for verifying that you're not looking at real-time faked face replacement won't hold up for much longer.

Don't forget what we're talking about, either. We're talking about accessing Wikimedia over HTTP, not briefing some official on top-secret information. How does "video interviews" solve "a highly distributed network of hacked machines is crawling my website using everyone's local identity"?

Re: Wikipedia is struggling with voracious AI bot crawlers

#88

Post by nonrandomstring »

concerndc1tizen wrote: post #61
nonrandomstring wrote: post #21

Earlier quoted context omitted.

> Why would you crawl the web interface when the data is so readily available in a even better format? To cause deliberate harm as a DDOS attack. Perhaps a better question is, why would companies who hope to replace human-curated static online information with their own generative service not use the cloak of "scraping" to take down their competition?

This is the most reasonable explanation. Wikipedia is openly opposed by the current US administration, and 'denial of service' is key to their strategy (i.e. tariffs, removal of rights/due process, breaking net neutrality, etc.). In the worst case, Wikipedia will have to require user login, which achieves the partial goal of making information inaccessible to the general public.

In the worst case Wikipedia will have to relocate to Europe and block the entire ASN of US network estates. But if the United States is determined to commit digital and economic suicide, I don't see how reasonable people can stop that.

Re: Wikipedia is struggling with voracious AI bot crawlers

#89

Post by microtherion »

PeterStuer wrote: post #81
microtherion wrote: post #14

Not just Wikipedia. My home server (hosting a number of not particularly noteworthy things, such as my personal gitea instance) has been absolutely hammered in recent months, to the extent of periodically bringing down the server for hours with thrashing. The worst part is that every single sociopathic company in the world seems to have simultaneously unleashed their own fleet of crawlers. Most of the bots downright…

Why not just rate limit? IP range based blocking will likely hit far more legitimate users than you think.

I know how I can block IPs in my router, but I'm not sure how I can rate limit. And I don't want to rate limit on the server level, because that in itself places load on the server.

Re: Wikipedia is struggling with voracious AI bot crawlers

#90

Post by concerndc1tizen »

nonrandomstring wrote: post #88
concerndc1tizen wrote: post #61

Earlier quoted context omitted.

This is the most reasonable explanation. Wikipedia is openly opposed by the current US administration, and 'denial of service' is key to their strategy (i.e. tariffs, removal of rights/due process, breaking net neutrality, etc.). In the worst case, Wikipedia will have to require user login, which achieves the partial goal of making information inaccessible to the general public.

In the worst case Wikipedia will have to relocate to Europe and block the entire ASN of US network estates. But if the United States is determined to commit digital and economic suicide, I don't see how reasonable people can stop that.

It would be trivial to use botnets inside of the EU, so I doubt that blocking ASNs would make any difference. And as I said, it achieves the goal of disrupting access to information, so that would be nevertheless be a win for them. Your proposition does not solve for Wikipedia's agenda of providing free access to information.

> digital and economic suicide

My view is that it's an economic coup which started decades ago (Bush-Halliburton, bank bailouts in 2008, etc.). It's only inflation and economic uncertainty is only for the poor. For the people that do algorithmic stock trading, it's an arbitrage opportunity that occurs in the scale of microseconds.

By the time that the people will be properly motivated to revolt against the government, it will be too late.

Post reply on HN