Tell HN: Amazonbot aggressively scraping my website and ignoring robots.txt

Post by schappim » Sun, Aug 02, 2026, 2:26 AM UTC

Post by schappim » Sun, Aug 02, 2026, 10:26 AM UTC

pera wrote: post #13

schappim wrote: post #12

It is not just Amazon (although they're #4 on my list of bad actors)[1]. 1. https://files.littlebird.com.au/bad-scrapers.png

Yeah it's not just Amazon, but so far Amazon was the only one specifically looking for URLs in source code. Interestingly it ignored URLs in the fake markdown docs. Another detail: the scraper did not attempt to access the endpoints immediately (as it did for hrefs in htmls) but it did it on the day after, twice.

Post by az09mugen » Sun, Aug 02, 2026, 10:46 AM UTC

Bender wrote: post #6

Assuming your site has no dependency on Amazon EC2: fetch_aws() { curl -s \ -H "Accept-Charset: UTF-8" \ -H "Accept-Encoding: gzip, deflate" \ -H "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8" \ -H "Accept-Language: en" \ -H "User-Agent: Mozilla/5.0 (Windows NT 11.1; rv:102.0; ) Gecko/20100101" \ -H "Cache-Control: 0" \ -H "Host: ip-ranges.amazonaws.com" \ -H "Connection: keep-alive" \ --re…

Post by cyanydeez » Sun, Aug 02, 2026, 8:30 PM UTC

schappim wrote: post #12

It is not just Amazon (although they're #4 on my list of bad actors)[1]. 1. https://files.littlebird.com.au/bad-scrapers.png

Post by rovr138 » Sun, Aug 02, 2026, 11:24 PM UTC

toomuchtodo wrote: post #5

pera wrote: post #4

Earlier quoted context omitted.

My concern is more in relation to trying to use an endpoint from non-public source code, it seems negligent to randomly try endpoints like this

Whether it’s negligent isn’t terribly relevant. Is it illegal? It is not, they aren’t bypassing access controls. No different than using Shodan, scanning public IPs, crawling open directories, etc. It would be different if they were attempting to brute force credentials to access an endpoint, but they aren’t.