Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenStreetMap overwhelmed by bots scraping data

twitter.com

Post reply on HN
Print view View original

1–10 of 24 posts

Re: OpenStreetMap overwhelmed by bots scraping data

#4

Post by Bender »

molly_radstowe wrote: post #2

#OpenStreetMap hammered by scrapers hiding behind residential proxy/embedded-SDK networks.

Looks like it is hosted in Equinix in NL? Or just part of it maybe? Is it behind a load balancer, maybe something like HAProxy? If so were stick tables set up to limit rates by cookie and require people be logged in on unique accounts and limit anonymous access after so many requests? I know limiting anonymous access is not great but that is something that could be enabled when under a high load so that instead of the site going offline for everyone it would just be limited for the anonymous users. Degradation vs critical outage

On a separate note have tcpdump captures been done on these excessive connections? Minus the IP, what do their SYN packets look like? Minus the IP what do the corresponding log entries look like in the web server? Are they using HTTP/1.1 or HTTP/2.0? Are they missing any expected headers for a real person such as cors, no-cors, navigate, accept_language?

    tcpdump -p --dont-verify-checksums -i any -NNnnvvv -B32768 -c32 -s0 port 443 and 'tcp[13] == 2'
Is there someone at OpenStreetMap that can answer these questions?

Re: OpenStreetMap overwhelmed by bots scraping data

#8

Post by KomoD »

Bender wrote: post #4
molly_radstowe wrote: post #2

#OpenStreetMap hammered by scrapers hiding behind residential proxy/embedded-SDK networks.

Looks like it is hosted in Equinix in NL? Or just part of it maybe? Is it behind a load balancer, maybe something like HAProxy? If so were stick tables set up to limit rates by cookie and require people be logged in on unique accounts and limit anonymous access after so many requests? I know limiting anonymous access is not great but that is something that could be enabled when under a high load so that instead of th…

I think it could be worth trying to block them with TLS fingerprinting, or since they think it's residential proxies they are being hammered by, https://spur.us could be worth a try.
Post reply on HN