Live data from Hacker News

Wikipedia is struggling with voracious AI bot crawlers

engadget.com

101–105 of 105 posts

Re: Wikipedia is struggling with voracious AI bot crawlers

#101
post #87

Earlier quoted context omitted.

That does not generically 100% solve the problem of "is this person a human". That ties things to an identity, but the verification that an identity is actually that human is not solved. Stolen identities, forged identities, faked identities, all still problems, and as soon as the full force of the black market capitalist market in such things is turned on that world, it'll still be a big problem. Also video-based ID…

>That does not generically 100% solve the problem of "is this person a human". you don't need 100% generic problem solvling. a "good enough solution" will block out 90% of low effort bad actors, and that's a huge relief by itself. That 9% will take some steps and be combatted, and that last 1% will never truly be held at bay.

Your model assumes the solutions aren't shared.

They are.

Hacker News readers tend to grotesquely underestimate the organization of the underworld, since they aren't in it and aren't generally affected by it. But the underworld is large and organized and very well funded. I'm sure you're operating on a mental model where everyone who sets out to scrape Wikimedia is some random hacker off on their own, sitting down to write a web scraper from scratch having never done it before and not being good at, and being just gobsmacked by the first anti-scraping tech they find, frantically searching online for how to bypass it and coming up with nothing.

That's not how the world works. Look around you, after all; you can already see the evidence of how untrue this is even in the garbage you find for yourself. You can see it in your spams, which never have problems finding hacked systems to put their forged login pages on. That's because the people sending the spam aren't hacking the systems themselves... they use a Hacked System As A Service provider. And I am not saying that sarcastically... that's exactly what they are. APIs and all. Bad actors do not sit down with a fresh college grad and a copy of Python for Dummies to write crawlers in the general case. (Some do, but honestly they're not the worrying ones.) They get Black Web Scraping As A Service, which is a company that can and does pay people full time to figure out how to get around blocks and limits, and when you see people asking questions about how to do that online, you're not seeing the pros. The pros don't ask those questions on Stack Exchange. They just consult their fellow employees, like any other business, because it's a business.

You could probably mentally model the collection of businesses I'm referring to as at least as large as Microsoft or Google, and generally staffed by people as intelligent.

It in fact does need to be a nearly 100% solution, because any crack will be found, exploited, and not merely "shared" but bought and sold freely, in a market that incentivizes people with big payouts to find the exploits.

I really wish people would understand this, the security defense team in the world is grotesquely understaffed and mismanaged collectively because people still think they're going up against some stereotypical sweaty guy in a basement who might get bored and wander away from hacking your site if he discovers women, rather than funded professionals attacking and spamming and infiltrating and getting paid large amounts of money to do it.

Re: Wikipedia is struggling with voracious AI bot crawlers

#102
post #31

Wikipedia spends 1% of its budget on hosting fees. It can spend a bit more given the rest of their corruptions.

Steady on cowboy. They might mislead the public in their needs and spends, but to jump from that to "corruption" is a bit too fast and loose. From their financial statement 2024 you can learn that they probably spend about $6,825,794 on site operation (excl. salaries etc.). This includes $3,116,445 for Internet hosting and an estimated $3,709,349 on Server infrastructure (est. as 85% of equipment). Now as of June 30,…

You say it’s not corruption but it’s still shady as hell.

I think it’s reasonable to say any shady stuff is a form of corruption

Re: Wikipedia is struggling with voracious AI bot crawlers

#103

Earlier quoted context omitted.

Steady on cowboy. They might mislead the public in their needs and spends, but to jump from that to "corruption" is a bit too fast and loose. From their financial statement 2024 you can learn that they probably spend about $6,825,794 on site operation (excl. salaries etc.). This includes $3,116,445 for Internet hosting and an estimated $3,709,349 on Server infrastructure (est. as 85% of equipment). Now as of June 30,…

You say it’s not corruption but it’s still shady as hell. I think it’s reasonable to say any shady stuff is a form of corruption

Shady as hell, sure. Misleading? At least misdirecting. But corruption? No.

If you have an olive green shirt and a grass green shirt, you can call both of them green, but calling the olive one grass green is a mistake.

Re: Wikipedia is struggling with voracious AI bot crawlers

#104

Earlier quoted context omitted.

You say it’s not corruption but it’s still shady as hell. I think it’s reasonable to say any shady stuff is a form of corruption

Shady as hell, sure. Misleading? At least misdirecting. But corruption? No. If you have an olive green shirt and a grass green shirt, you can call both of them green, but calling the olive one grass green is a mistake.

I think you need to think carefully as to what corruption means. I would go so far as to say man's inability to recognise things as corrupt is a corruption of man.

Re: Wikipedia is struggling with voracious AI bot crawlers

#105

Earlier quoted context omitted.

You don't need a VC if your sitting on $415,9 Million.

They didn't start with $415M, and they didn't take VC money to get there either -- that's the point.

Yes, they didn't take wealthy person VC money, but rather begged for $2 donations from people who actually need the money.
Post reply on HN