A year of fighting scrapers on my 1.5 million-page website
1–10 of 449 posts
Re: A year of fighting scrapers on my 1.5 million-page website
#2[flagged]
Re: A year of fighting scrapers on my 1.5 million-page website
#3[flagged]
what gave you this idea?
Re: A year of fighting scrapers on my 1.5 million-page website
#4> And yes, my site gets its data by scraping those public documents. So I'm a scraper writing a blog post complaining about scrapers. I'm aware of how that sounds.
Re: A year of fighting scrapers on my 1.5 million-page website
#5Re: A year of fighting scrapers on my 1.5 million-page website
#6blocking by geo yeah might work - but what happens when someone is traveling abroad ? they've to use a VPN to access your site ?
my take with all the bots - the web is gonna be a bunch of private walled gardens. with most sites set to no index. you will only discover them via referral from someone real.
Re: A year of fighting scrapers on my 1.5 million-page website
#7Re: A year of fighting scrapers on my 1.5 million-page website
#8pretty crazy that this article is seemingly written by an AI used a detector and it is 95% confident its generated
Re: A year of fighting scrapers on my 1.5 million-page website
#9Is there any cheap way to run personal websites without getting cooked by bots these days, degrading the performance? A $5/month VPS won't cut it anymore.
Re: A year of fighting scrapers on my 1.5 million-page website
#10I'm in a similar boat. Probably 99.999% is bots. I have nearly 1m unique "visitors" according to cloudflare and my real users are in the dozens a day. That being said, I love the open internet and am holding on to keeping as much open as I can.