We host a web app which has around 50,000 statically generated public pages, and the amount of bot traffic is insane. Facebook’s crawler requested the same pages over 3 million times in 2 days, and we’ve used Vercel and Cloudflare to block as much as possible but it’s not working. The AI scrapers are routing requests through residential proxies, and even after putting our entire app behind an Auth gate with cloudflare turnstile they’re creating accounts to get access to data that’s meant for humans to read / use. I’ve never seen anything like it before, our hosting costs are up like crazy, and besides all of this, I can’t trust anything I read or see on this web anymore. As a last resort we’ll try switching to hcaptcha tomorrow instead of turnstile which appears to be more difficult but this is all so depressing to me. Want to hear if anyone else is experiencing this, there’s no way I’m the only here who is struggling with sophisticated bots?
Ask HN: The amount of AI bot traffic is out control?
1–7 of 7 posts
Re: Ask HN: The amount of AI bot traffic is out control?
#2How do you know it's facebook's crawler if it going through residential proxies?
Re: Ask HN: The amount of AI bot traffic is out control?
#3How do you know it's facebook's crawler if it going through residential proxies?
It had meta in the user-agent headers, it’s trivial to spoof but I’ve read of many others who are seeing the same thing. It’s not just them though, it’s also parallel web systems, and then a bunch of others who aren’t identifying themselves.
Re: Ask HN: The amount of AI bot traffic is out control?
#4[flagged]
Re: Ask HN: The amount of AI bot traffic is out control?
#5I doubt it's really Facebook creating fake accounts and using proxies, why would they keep their real UA then?
Anyhow, I occasionally suffer this same problem when happy crawlers find my site and use truesign.ai to block them, so far successfully. It detects residential proxies, fake emails and overall suspicious activity, and - something I really wanted to avoid - there are no captchas.
Re: Ask HN: The amount of AI bot traffic is out control?
#6How do you know it's facebook's crawler if it going through residential proxies?
It had meta in the user-agent headers, it’s trivial to spoof but I’ve read of many others who are seeing the same thing. It’s not just them though, it’s also parallel web systems, and then a bunch of others who aren’t identifying themselves.
That's probably not Meta themselves, but their new agent called Muse. It's likely coming from actual individuals. Not sure why they're scraping your website though. Maybe time to turn off all the SEO stuff to make your site less discoverable?
Re: Ask HN: The amount of AI bot traffic is out control?
#7> The AI scrapers are routing requests through residential proxies
How do you differentiate between an Ai scraper and an end user having an agent perform a search and then fetch all the results to analyze them for relevance? (because search has become so bad I need an agent to deal with it before I look at things)