Earlier quoted context omitted.
Whether it’s negligent isn’t terribly relevant. Is it illegal? It is not, they aren’t bypassing access controls. No different than using Shodan, scanning public IPs, crawling open directories, etc. It would be different if they were attempting to brute force credentials to access an endpoint, but they aren’t.
> they aren’t bypassing access controls. Weev went to jail for accessing public api's, https://en.wikipedia.org/wiki/Weev#AT&T_data_breach > The flaw was part of a publicly-accessible URL, which allowed the group to collect the e-mails without having to break into AT&T's system. It was argued that he didn't circumvent, but it didn't stop them from putting him in jail initially.
Tell HN: Amazonbot aggressively scraping my website and ignoring robots.txt
21–26 of 26 posts
Re: Tell HN: Amazonbot aggressively scraping my website and ignoring robots.txt
#22Re: Tell HN: Amazonbot aggressively scraping my website and ignoring robots.txt
#23Earlier quoted context omitted.
Perhaps you need a TOS that requires a usage fee be mailed to a PO Box that has a reasonable "free" limit of like 100$ and when they exceed it, you automatically mail them a copy of the TOS, the logs and an invoice. Just make sure you have a good lawyer.
is there a law they’re breaking? because idk i could be wrong but some small project vs a 2.5T market cap company is gonna need more than “a good lawyer”
Re: Tell HN: Amazonbot aggressively scraping my website and ignoring robots.txt
#24Earlier quoted context omitted.
Perhaps you need a TOS that requires a usage fee be mailed to a PO Box that has a reasonable "free" limit of like 100$ and when they exceed it, you automatically mail them a copy of the TOS, the logs and an invoice. Just make sure you have a good lawyer.
is there a law they’re breaking? because idk i could be wrong but some small project vs a 2.5T market cap company is gonna need more than “a good lawyer”
In the UK, you would likely win in court, and be awarded £1 for your increased hosting costs. There isn't the concept of punitive damages for contract law here. YMMV.
However. People report that you can get paid by sending a plausible invoice to a tech company. The culture disincentivises verifying purchase orders.
Re: Tell HN: Amazonbot aggressively scraping my website and ignoring robots.txt
#25Meta not only ignored my robots.txt but they hit me with 1k requests per second sometimes.
BingBot is doing something weird. Based on the logs I see, it feels like their crawler is trying to reverse engineer my public facing APIs and scrape those directly.