271–280 of 466 posts
Seems like many of these "AI companies" wouldn't need another funding round if they would do scraping ... (ironically) more intelligently. Really, this behaviour should be a big embarrassment for any company whose main business model is selling "intelligence" as an outside product.
Is any on them even close to profitable?
Earlier quoted context omitted.
Run a mediawiki, as described in the post. It's very heavy. Specifically for history I'm guessing it has to re-parse the entire page and do all link and template lookups because previous versions of the page won't be in any cache
The original post says it's not actually a burden though; they just don't like it. If something is so heavy that 2 requests/second matters, it would've been completely infeasible in say 2005 (e.g. a low power n100 is ~20x faster than the athlon xp 3200+ I used back then. An i5-12600 is almost 100x faster. Storage is >1000x faster now). Or has mediawiki been getting less efficient over the years to keep up with more p…
> And I mean that - they indexed every single diff on every page for every change ever made. Frequently with spikes of more than 10req/s. Of course, this made MediaWiki and my database server very unhappy, causing load spikes, and effective downtime/slowness for the human users.
Earlier quoted context omitted.
> Cloudflare also has a feature to block known AI bots and even suspected AI bots In addition to other crushing internet risks, add wrongly blacklisted as a bot to the list.
These features are opt-in and often paid features. I struggle to see how this is a "crushing risk," although I don't doubt that sufficiently unskilled shops would be completely crushed by an IP/userAgent block. Since Cloudflare has a much more informed and broader view of internet traffic than maybe any other company in the world, I'll probably use that feature without any qualms at some point in the future. Right no…
Earlier quoted context omitted.
https://web.archive.org/web/20240101000000*/https://wiki.dia...
notice how there's a period of almost two months with no new index, just until a week before I posted this? I wonder what might have caused this!!1 (and it's not like they only check robots.txt once a month or so. https://stuff.overengineer.dev/stash/2024-12-30-dfwiki-opena... )
Earlier quoted context omitted.
I'd kind of like to see that claim substantiated a little more. Is it all crawlers that switch to a non-bot UA, or how are they determining it's the same bot? What non-bot UA do they claim?
> Is it all crawlers that switch to a non-bot UA I've observed only one of them do this with high confidence. > how are they determining it's the same bot? it's fairly easy to determine that it's the same bot, because as soon as I blocked the "official" one, a bunch of AWS IPs started crawling the same URL patterns - in this case, mediawiki's diff view (`/wiki/index.php?title=[page]&diff=[new-id]&oldid=[old-id]`), th…
Earlier quoted context omitted.
The original post says it's not actually a burden though; they just don't like it. If something is so heavy that 2 requests/second matters, it would've been completely infeasible in say 2005 (e.g. a low power n100 is ~20x faster than the athlon xp 3200+ I used back then. An i5-12600 is almost 100x faster. Storage is >1000x faster now). Or has mediawiki been getting less efficient over the years to keep up with more p…
Oh, I was a bit off. They also indexed diffs > And I mean that - they indexed every single diff on every page for every change ever made. Frequently with spikes of more than 10req/s. Of course, this made MediaWiki and my database server very unhappy, causing load spikes, and effective downtime/slowness for the human users.
For what it's worth my kiwix copy of Wikipedia has a ~5ms response time for an uncached article according to Firefox. If I hit a single URL with wrk (so some caching at least with disks. Don't know what else kiwix might do) at concurrency 8, it does 13k rps on my n305 with a 500 us average response time. That's over 20Gbit/s, so basically impossible to actually saturate. If I load test from another computer it uses ~0.2 cores to max out 1Gbit/s. Different code bases and presumably kiwix is a bit more static, but at least provides a little context to compare with for orders of magnitude. A 3 OOM difference seems pretty extreme.
Incidentally, local copies of things are pretty great. It really makes you notice how slow the web is when links open in like 1 frame.
- detect bot IPs, serve them special pages
- special pages require javascript to render
- javascript mines bitcoin
- result of mining gets back to your server somehow (encoded in which page they fetch next?)Earlier quoted context omitted.
The linked article explains what happens when you block their IP.
For reference: > If you try to rate-limit them, they’ll just switch to other IPs all the time. If you try to block them by User Agent string, they’ll just switch to a non-bot UA string (no, really). It's really absurd that they seem to think this is acceptable.
Earlier quoted context omitted.
What do you mean crushing risk? Just solve these 12 puzzles by moving tiny icons on tiny canvas while on the phone and you are in the clear for a couple more hours!
Sometimes it doesn’t even give you a Captcha. I have come across some websites that block me using Cloudflare with no way of solving it. I’m not sure why, I’m in a large first-world country, I tried a stock iPhone and a stock Windows PC, no VPN or anything. That’s just no way to know.
Earlier quoted context omitted.
What do you use to block them?
Nginx, it's nothing special it's just my load balancer. if ($http_user_agent ~* (list|of|case|insensitive|things|to|block)) {return 403;}