Live data from Hacker News

Tell HN: Amazonbot aggressively scraping my website and ignoring robots.txt

news.ycombinator.com

21–26 of 26 posts

Re: Tell HN: Amazonbot aggressively scraping my website and ignoring robots.txt

#21
post #20

Earlier quoted context omitted.

Whether it’s negligent isn’t terribly relevant. Is it illegal? It is not, they aren’t bypassing access controls. No different than using Shodan, scanning public IPs, crawling open directories, etc. It would be different if they were attempting to brute force credentials to access an endpoint, but they aren’t.

> they aren’t bypassing access controls. Weev went to jail for accessing public api's, https://en.wikipedia.org/wiki/Weev#AT&T_data_breach > The flaw was part of a publicly-accessible URL, which allowed the group to collect the e-mails without having to break into AT&T's system. It was argued that he didn't circumvent, but it didn't stop them from putting him in jail initially.

His conviction was vacated and Amazon has deep pockets and diffusion of internal liability. The illicit state drug charges did not help his case.

Re: Tell HN: Amazonbot aggressively scraping my website and ignoring robots.txt

#23

Earlier quoted context omitted.

Perhaps you need a TOS that requires a usage fee be mailed to a PO Box that has a reasonable "free" limit of like 100$ and when they exceed it, you automatically mail them a copy of the TOS, the logs and an invoice. Just make sure you have a good lawyer.

is there a law they’re breaking? because idk i could be wrong but some small project vs a 2.5T market cap company is gonna need more than “a good lawyer”

Edit: oops, double-posted.

Re: Tell HN: Amazonbot aggressively scraping my website and ignoring robots.txt

#24

Earlier quoted context omitted.

Perhaps you need a TOS that requires a usage fee be mailed to a PO Box that has a reasonable "free" limit of like 100$ and when they exceed it, you automatically mail them a copy of the TOS, the logs and an invoice. Just make sure you have a good lawyer.

is there a law they’re breaking? because idk i could be wrong but some small project vs a 2.5T market cap company is gonna need more than “a good lawyer”

It would fall under contact law, and the case would hinge on either the ToS is binding, and whether the bot operator is deemed to have accepted the ToS, and whether access inflicts a cost.

In the UK, you would likely win in court, and be awarded £1 for your increased hosting costs. There isn't the concept of punitive damages for contract law here. YMMV.

However. People report that you can get paid by sending a plausible invoice to a tech company. The culture disincentivises verifying purchase orders.

Re: Tell HN: Amazonbot aggressively scraping my website and ignoring robots.txt

#25
It's not abnormal for people to disrespect robots.txt

Meta not only ignored my robots.txt but they hit me with 1k requests per second sometimes.

BingBot is doing something weird. Based on the logs I see, it feels like their crawler is trying to reverse engineer my public facing APIs and scrape those directly.

Post reply on HN