Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Website with 6^16 subpages and 80k+ daily bots

news.ycombinator.com

Post reply on HN
Print view View original

71–80 of 212 posts

Re: Ask HN: Website with 6^16 subpages and 80k+ daily bots

#71

Post by imdsm »

tomcam wrote: post #44
cookiengineer wrote: post #36

First off, make a website defend mode that can be triggered to serve different content. Then, do the following: 1. Add a robots.txt, make it look like it's wordpress (Disallow: /wp-admin etc) 2. If any client requests /wp-admin, flag their IP ASN as bot. 3. If a client is a bot, send it a gzip bomb (100kB size, unpacked around 20GB), use Transfer-Encoding: gzip and Content-Encoding: gzip to also punish malicious web…

I would like to be your friend for 2 reasons. #1 is that you’re brilliantly devious. #2 is that I fervently wish to stay on your good side.

I too wish to join this group

Re: Ask HN: Website with 6^16 subpages and 80k+ daily bots

#73

Post by tmountain »

cookiengineer wrote: post #36

First off, make a website defend mode that can be triggered to serve different content. Then, do the following: 1. Add a robots.txt, make it look like it's wordpress (Disallow: /wp-admin etc) 2. If any client requests /wp-admin, flag their IP ASN as bot. 3. If a client is a bot, send it a gzip bomb (100kB size, unpacked around 20GB), use Transfer-Encoding: gzip and Content-Encoding: gzip to also punish malicious web…

I come to HN every day just hoping to stumble onto these kinds of gems. You, sir, are fighting the good fight! ;-)

Re: Ask HN: Website with 6^16 subpages and 80k+ daily bots

#74

Post by mrtksn »

cookiengineer wrote: post #36

First off, make a website defend mode that can be triggered to serve different content. Then, do the following: 1. Add a robots.txt, make it look like it's wordpress (Disallow: /wp-admin etc) 2. If any client requests /wp-admin, flag their IP ASN as bot. 3. If a client is a bot, send it a gzip bomb (100kB size, unpacked around 20GB), use Transfer-Encoding: gzip and Content-Encoding: gzip to also punish malicious web…

The gzip idea is giving me goosebumps however this must be a solved problem, right? I mean, the client device can also send zip bombs so it sounds like it should be DDOS 101?

Re: Ask HN: Website with 6^16 subpages and 80k+ daily bots

#75

Post by Gud »

cookiengineer wrote: post #36

First off, make a website defend mode that can be triggered to serve different content. Then, do the following: 1. Add a robots.txt, make it look like it's wordpress (Disallow: /wp-admin etc) 2. If any client requests /wp-admin, flag their IP ASN as bot. 3. If a client is a bot, send it a gzip bomb (100kB size, unpacked around 20GB), use Transfer-Encoding: gzip and Content-Encoding: gzip to also punish malicious web…

Thanks a lot for the friendly advice. I’ll check your GitHub for sure.

Re: Ask HN: Website with 6^16 subpages and 80k+ daily bots

#76

Post by b112 »

poincaredisk wrote: post #67
b112 wrote: post #55

Earlier quoted context omitted.

Only someone poking about would ever hit that url on someone else's domain, so where's the downside? And "a lot" of false positives?? Recall, robots.txt is set to ignore this, so only malicious web scanners will hit it.

The downside is that you ban a whole ISP because of a single user misbehaving. Personally I sometimes do a quick request to /wp-admin to check if a site is WordPress, so I guess that has a nonzero chance of affecting me. And when I mirror a website I almost always ignore robots.txt (I'm not a robot and I do it for myself). And when I randomly open robots.txt and see a weird url I often visit it. And these are just my…

Well you make a point, I use ipset in many circumstances, which has an expire option.

So that is a balance between a bad actor and even "stop it" blocks, and auto expire means transitory denial.

Re: Ask HN: Website with 6^16 subpages and 80k+ daily bots

#77

Post by 2000swebgeek »

PeterStuer wrote: post #51
cookiengineer wrote: post #36

First off, make a website defend mode that can be triggered to serve different content. Then, do the following: 1. Add a robots.txt, make it look like it's wordpress (Disallow: /wp-admin etc) 2. If any client requests /wp-admin, flag their IP ASN as bot. 3. If a client is a bot, send it a gzip bomb (100kB size, unpacked around 20GB), use Transfer-Encoding: gzip and Content-Encoding: gzip to also punish malicious web…

"If any client requests /wp-admin, flag their IP ASN as bot" You are going to hit a lot more false positives with this one than actual bots

better yet, see if bots access /robots.txt, find them from there. no human looks at robots.txt :)

add a captcha by limiting IP requests or return 429 to rate limit by IP. Using popular solutions like cloudflare could help reduce the load. Restrict by country. Alternatively, put in a login page which only solves the captcha and issues a session.

Re: Ask HN: Website with 6^16 subpages and 80k+ daily bots

#78

Post by ta12653421 »

cookiengineer wrote: post #36

First off, make a website defend mode that can be triggered to serve different content. Then, do the following: 1. Add a robots.txt, make it look like it's wordpress (Disallow: /wp-admin etc) 2. If any client requests /wp-admin, flag their IP ASN as bot. 3. If a client is a bot, send it a gzip bomb (100kB size, unpacked around 20GB), use Transfer-Encoding: gzip and Content-Encoding: gzip to also punish malicious web…

++1

real pro, man, wow! :))

Re: Ask HN: Website with 6^16 subpages and 80k+ daily bots

#79

Post by j-a-a-p »

dankwizard wrote: post #34

Sell it to someone inexperienced who wants to pick up a high traffic website. Show the stats of visitors, monthly hits, etc. DO NOT MENTION BOTS. Easiest money you'll ever make. (Speaking from experience ;) )

> (Speaking from experience ;) )

Assuming you were the buyer. Did they also ran an add clicking scheme on it, to even further inflate their profit?

https://github.com/CryptoidCoder/Selenium-Ad-Clicker

Re: Ask HN: Website with 6^16 subpages and 80k+ daily bots

#80

Post by mkl »

damir wrote: post #14
nojvek wrote: post #13

Earlier quoted context omitted.

Not really. When numbers repeat, the value is the same. E.g 00 is the same as 00. So the possible outcomes is 6^16, but unique values per color channel is only 256 values. So unique colors are 256^3 = 16.7M colors.

Yes, each possible 6^16 outcome is it's own subpage... /000000 /000001 /000002 /000003 etc... Or am I missing something?

You are missing something. How many two-digit decimal numbers are there from 00 to 99? Obviously 99+1 = 100: 10 options for the first digit times 10 options for the second digit; 10 in the form 0X, 10 in the form 1X, etc. up to 9X, a total of 10 * 10 = 10^2.

So how many 6-digit hexadecimal numbers from 0x000000 to 0xffffff? 0xffffff+1 = 16777216 = 16^6. 16 options for the first digit, times 16 options for the second digit, times 16 for the 3rd, times 16 for the 4th, times 16 for the 5th, times 16 for the 6th is 16^6. Or go to bytes: 3 bytes, each with 256 possible values is 256^3 = (16^2)^3 = 16^6. Or bits: 2^24 = (2^4)^6 = 16^6.

It's also pretty trivial to just count them. Run this in your browser console:

  count = 0; for(i = 0x000000; i
Post reply on HN