Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Website with 6^16 subpages and 80k+ daily bots

news.ycombinator.com

Post reply on HN
Print view View original

91–100 of 212 posts

Re: Ask HN: Website with 6^16 subpages and 80k+ daily bots

#91

Post by brazzy »

yjftsjthsd-h wrote: post #40
chirau wrote: post #38

Earlier quoted context omitted.

Interesting. What does number 5 do? Also, how do gzip bombs works, does it automatically extract to the 20gb or the bot has to initiate the extraction?

Yes, it requires the client to try and extract the archive; https://en.wikipedia.org/wiki/Zip_bomb is the generic description.

What archive? The idea was to use Transfer-Encoding: gzip, which means the compression is a transparent part of the HTTP request which the client HTTP library will automatically try to extract.

Re: Ask HN: Website with 6^16 subpages and 80k+ daily bots

#92

Post by str3wer »

PeterStuer wrote: post #62
b112 wrote: post #55

Earlier quoted context omitted.

Only someone poking about would ever hit that url on someone else's domain, so where's the downside? And "a lot" of false positives?? Recall, robots.txt is set to ignore this, so only malicious web scanners will hit it.

Do you own your ASN or unique IP? Do you like getting banned for the actions of others that share your ASN or IP?

what chance are we even talking of a false positive?

Re: Ask HN: Website with 6^16 subpages and 80k+ daily bots

#93

Post by mrtksn »

chgs wrote: post #90
mrtksn wrote: post #86

Earlier quoted context omitted.

What if they send a POST request?

Most servers will limit posts to a fairly low size by default.

Yes but the idea behind zip bombs that they appear to be very small, when expanded it can be extremely large. Before attempting to decompress, the POST request may appear something like 20kb and end up being 20gb.

Re: Ask HN: Website with 6^16 subpages and 80k+ daily bots

#94

Post by Aeolun »

discoinverno wrote: post #59
cookiengineer wrote: post #36

First off, make a website defend mode that can be triggered to serve different content. Then, do the following: 1. Add a robots.txt, make it look like it's wordpress (Disallow: /wp-admin etc) 2. If any client requests /wp-admin, flag their IP ASN as bot. 3. If a client is a bot, send it a gzip bomb (100kB size, unpacked around 20GB), use Transfer-Encoding: gzip and Content-Encoding: gzip to also punish malicious web…

Unrelated, but if I try to send you a message on https://cookie.engineer/contact.html it says "Could not send message, check ad-blocking extension", but I'm pretty sure I turned them off and it still doesn't work Also, the best starter is charmender

Mander! Best pokemon, but can’t write the name.

Also, the best one is clearly Squirtle.

Re: Ask HN: Website with 6^16 subpages and 80k+ daily bots

#95

Post by hypeatei »

notachatbot123 wrote: post #63
PeterStuer wrote: post #60

Earlier quoted context omitted.

If you ban an IP or even an ASN, there could be (many) thousands sharing that same identifier. Some kid will unknowingly run some free game that does some lightweight scraping in the background as monetization and you ban the whole ISP?

> some free game that does some lightweight scraping in the background as monetization What in the flying **. Is this a common thing?

Residential IPs are extremely valuable for scraping or other automation flows so yeah getting kids to run a free game that has malware seems plausible.

Re: Ask HN: Website with 6^16 subpages and 80k+ daily bots

#96

Post by Aeolun »

mrtksn wrote: post #93
chgs wrote: post #90

Earlier quoted context omitted.

Most servers will limit posts to a fairly low size by default.

Yes but the idea behind zip bombs that they appear to be very small, when expanded it can be extremely large. Before attempting to decompress, the POST request may appear something like 20kb and end up being 20gb.

I’d have to assume a decent implementation will eventually give up. It’s almost a given those will not be used by people crawling the internet for quick wins though.

Re: Ask HN: Website with 6^16 subpages and 80k+ daily bots

#97

Post by Aeolun »

TrainedMonkey wrote: post #53
cookiengineer wrote: post #36

First off, make a website defend mode that can be triggered to serve different content. Then, do the following: 1. Add a robots.txt, make it look like it's wordpress (Disallow: /wp-admin etc) 2. If any client requests /wp-admin, flag their IP ASN as bot. 3. If a client is a bot, send it a gzip bomb (100kB size, unpacked around 20GB), use Transfer-Encoding: gzip and Content-Encoding: gzip to also punish malicious web…

Is this strictly legal? For example, in the scenario where a "misconfigured" bot of a large evil corporation get's taken down and, due to layers of ass covering, they think it's your fault and it cost them a lot of money. Do they have a legal case that could fly in eastern district of Texas?

I don’t think there is anything illegal about serving a large payload? If they dob’t like it they can easily stop making requests.

Re: Ask HN: Website with 6^16 subpages and 80k+ daily bots

#98

Post by codingdave »

PeterStuer wrote: post #60
afandian wrote: post #54

Earlier quoted context omitted.

Why? Who is legitimately going to that address but the site admin?

If you ban an IP or even an ASN, there could be (many) thousands sharing that same identifier. Some kid will unknowingly run some free game that does some lightweight scraping in the background as monetization and you ban the whole ISP?

How would that be a false positive? The kid might not be malicious, but they absolutely are running a bot, even if unknowingly. If anything, calling attention to it could help people notice, and therefore clean up such things.

Re: Ask HN: Website with 6^16 subpages and 80k+ daily bots

#99

Post by Ylpertnodi »

TrainedMonkey wrote: post #53
cookiengineer wrote: post #36

First off, make a website defend mode that can be triggered to serve different content. Then, do the following: 1. Add a robots.txt, make it look like it's wordpress (Disallow: /wp-admin etc) 2. If any client requests /wp-admin, flag their IP ASN as bot. 3. If a client is a bot, send it a gzip bomb (100kB size, unpacked around 20GB), use Transfer-Encoding: gzip and Content-Encoding: gzip to also punish malicious web…

Is this strictly legal? For example, in the scenario where a "misconfigured" bot of a large evil corporation get's taken down and, due to layers of ass covering, they think it's your fault and it cost them a lot of money. Do they have a legal case that could fly in eastern district of Texas?

Texas, America?
Post reply on HN