301–310 of 422 posts
Earlier quoted context omitted.
I doubt they are doing it. You just have to get on a VPN to and see yourself being flooded with captchas despite browsing the web like a normal human and solving dozens of captchas along the way.
I guess that raises the bot score given that the vpn IP is a data Center IP and thus in a lot of ban lists
https://www.kaspersky.com/blog/what-is-wrong-with-free-vpn-s...
Earlier quoted context omitted.
> Unfortunately whatever HN is using routinely blocks my login with "Sorry." I believe that's the HN application itself, not a WAF in front of it.
HN is surprisingly very very guilty of a whole lot of anti-user patterns and behaviour that other companies get regularly lamented. Poor accessibility, bad mobile support, no options to delete content beyond a narrow window.
Poor accessibility
> ...we have tried to minimize the impact on real readers as much as possible. We have not gone with tools like Anubis, partly because it causes annoying delays for those trying to get to the site, but also partly because it seems inevitable that the scrapers will eventually find their way around it. Indeed, there are some indications that is already happening. A proof-of-work requirement is not a huge obstacle when…
> The dilemma for bots: when tokens are bound to the connecting ip, scrapers must limit the connecting IP pool for each site they want to scrape, becoming much more obvious and easy to block, or they have to use massive amounts of compute. You can't do that any more. Too many ISPs, especially mobile carriers, don't hand out anything resembling a fixed IP address any more. It's CGNAT and constantly changing IP address…
Private trackers do this. If they ban a user that geolocates to a certain city and ISP, they'll ban new signups from that city and ISP because there's probably only a few users from the same city and ISP. And then report to their friends at other trackers, that a user with that city and ISP is trying to evade a ban.
>There are ways to tell the difference — the bots usually do not fetch images or CSS, for example — but, by the time that determination is made, the address in question will not be used again. Blocking the address at that point is just a waste of time. I don't get it. Don't we keep blacklists of this stuff? And if they hammer thousands of requests per site per second and never reuse an IP, they'd run out of addresses…
> do you want random IoT devices talking to your web server? Probably not, but since IoT manufacturers did zero to lock down their devices, those devices are doing a lot more than their owners think they are doing
Earlier quoted context omitted.
I agree, if up-to-data data was available somewhere else and free, there would be no reason to pay hackers and scrape. You could perhaps even get website operators to "push" new data to a common crawl database. The scrapers would learn there is no value on scraping X domain because the data is available elsewhere more easily.
Well this is not what is happening in practice, Wikipedia / Wikidata, OpenStreetMap, OpenFoodFacts... All provide APIs and even a full dump of their database available to download for free, but no, the stupid bots still DDoS them 24h/24.
> There are ways to tell the difference — the bots usually do not fetch images or CSS, for example — but, by the time that determination is made, the address in question will not be used again. Blocking the address at that point is just a waste of time. Maybe there's no point for the scanned server to block the address, but couldn't collective / shared block lists help with sites that may get scanned by the same addr…
The users presumably don't know about this, or you know, they clicked, "I agree."
Nearly Half of LG Smart TV Apps Contain Residential Proxies
> There are ways to tell the difference — the bots usually do not fetch images or CSS, for example — but, by the time that determination is made, the address in question will not be used again. Blocking the address at that point is just a waste of time. Maybe there's no point for the scanned server to block the address, but couldn't collective / shared block lists help with sites that may get scanned by the same addr…
At which point, millions of people will be forced to complain to their local representatives and... hey presto? :)
> ...we have tried to minimize the impact on real readers as much as possible. We have not gone with tools like Anubis, partly because it causes annoying delays for those trying to get to the site, but also partly because it seems inevitable that the scrapers will eventually find their way around it. Indeed, there are some indications that is already happening. A proof-of-work requirement is not a huge obstacle when…
Anubis is by far the least annoying throttler I encounter. Entirely agreed, just crank it up when you get a flood, I much prefer waiting a couple seconds to interacting with custom UI for tens of seconds. I'm so glad to see that (essentially) HashCash is coming back. Now we just need it for email, like it was originally designed for...
A few months ago there was a story posted here about someone who completely eliminated crawlers on their website with Anubis.
I think it was getting upvoted before users were clicking the article because if you did, you had to leave the Anubis PoW page open for several minutes before you could get into the site. The Anubis difficulty scale is unintuitive and the difference between a small delay and becoming unusable is easy to cross.
Earlier quoted context omitted.
I looked this up and realised it’s the page I’d seen briefly on a range of websites lately. It’s not annoyed me at all. Not nearly as much as having to complete captchas with slow refreshing tiles.
I've usually been more annoyed at the surprise dissonance of "was that an anime girl on kernel.org?" than annoyed at the delay.
Earlier quoted context omitted.
> The anubis author has stated they recognize it's an arms race, but PoW scales. The scraper wars are largely between script kiddies and people with both deep intimate networking and DOM knowledge. Yes greyhairs, I’m looking at you. The problem is, you can’t PoW every page load and resource request because the user experience will suck and people will run away. And that window - the gap between what people will toler…
Oh, but you can PoW every page. Your typical end user doesn't switch IPs that often, so it's fine to Anubis them again when they do. A scraper, on the other hand, has a tradeoff to make between rotating ips often (requiring a challenge on every request) or keeping only a few IPs (making cross-request identification much more valuable and reliable).
They meant you can’t PoW every page transition.
If clicking every link on your website throws you back to another Anubis page for 2-3 seconds, users will bounce.
That’s why Anubis does an up front challenge and then you’re good for a while. It’s a really low cost for the scrapers.