Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Website with 6^16 subpages and 80k+ daily bots

news.ycombinator.com

Post reply on HN
Print view View original

51–60 of 212 posts

Re: Ask HN: Website with 6^16 subpages and 80k+ daily bots

#51

Post by PeterStuer »

cookiengineer wrote: post #36

First off, make a website defend mode that can be triggered to serve different content. Then, do the following: 1. Add a robots.txt, make it look like it's wordpress (Disallow: /wp-admin etc) 2. If any client requests /wp-admin, flag their IP ASN as bot. 3. If a client is a bot, send it a gzip bomb (100kB size, unpacked around 20GB), use Transfer-Encoding: gzip and Content-Encoding: gzip to also punish malicious web…

"If any client requests /wp-admin, flag their IP ASN as bot"

You are going to hit a lot more false positives with this one than actual bots

Re: Ask HN: Website with 6^16 subpages and 80k+ daily bots

#52

Post by nneonneo »

scrps wrote: post #25

Clearly adjust glasses as an HN amateur color theorist[1] I am shocked and quite frankly appalled that you wouldn't also link to LAB, HSV, and CMYK equivalents, individually of course! /s That should generate you some link depth for the bots to burn cycles and bandwidth on. [1]: Not even remotely a color theorist

What you really should do is have floating point subpages for giggles, like /LAB/0.317482834/0.8474728828/0.172737838. Then you can have a literally infinite number of pages!

Re: Ask HN: Website with 6^16 subpages and 80k+ daily bots

#53

Post by TrainedMonkey »

cookiengineer wrote: post #36

First off, make a website defend mode that can be triggered to serve different content. Then, do the following: 1. Add a robots.txt, make it look like it's wordpress (Disallow: /wp-admin etc) 2. If any client requests /wp-admin, flag their IP ASN as bot. 3. If a client is a bot, send it a gzip bomb (100kB size, unpacked around 20GB), use Transfer-Encoding: gzip and Content-Encoding: gzip to also punish malicious web…

Is this strictly legal? For example, in the scenario where a "misconfigured" bot of a large evil corporation get's taken down and, due to layers of ass covering, they think it's your fault and it cost them a lot of money. Do they have a legal case that could fly in eastern district of Texas?

Re: Ask HN: Website with 6^16 subpages and 80k+ daily bots

#54

Post by afandian »

PeterStuer wrote: post #51
cookiengineer wrote: post #36

First off, make a website defend mode that can be triggered to serve different content. Then, do the following: 1. Add a robots.txt, make it look like it's wordpress (Disallow: /wp-admin etc) 2. If any client requests /wp-admin, flag their IP ASN as bot. 3. If a client is a bot, send it a gzip bomb (100kB size, unpacked around 20GB), use Transfer-Encoding: gzip and Content-Encoding: gzip to also punish malicious web…

"If any client requests /wp-admin, flag their IP ASN as bot" You are going to hit a lot more false positives with this one than actual bots

Why? Who is legitimately going to that address but the site admin?

Re: Ask HN: Website with 6^16 subpages and 80k+ daily bots

#55

Post by b112 »

PeterStuer wrote: post #51
cookiengineer wrote: post #36

First off, make a website defend mode that can be triggered to serve different content. Then, do the following: 1. Add a robots.txt, make it look like it's wordpress (Disallow: /wp-admin etc) 2. If any client requests /wp-admin, flag their IP ASN as bot. 3. If a client is a bot, send it a gzip bomb (100kB size, unpacked around 20GB), use Transfer-Encoding: gzip and Content-Encoding: gzip to also punish malicious web…

"If any client requests /wp-admin, flag their IP ASN as bot" You are going to hit a lot more false positives with this one than actual bots

Only someone poking about would ever hit that url on someone else's domain, so where's the downside?

And "a lot" of false positives?? Recall, robots.txt is set to ignore this, so only malicious web scanners will hit it.

Re: Ask HN: Website with 6^16 subpages and 80k+ daily bots

#56

Post by Havoc »

dankwizard wrote: post #34

Sell it to someone inexperienced who wants to pick up a high traffic website. Show the stats of visitors, monthly hits, etc. DO NOT MENTION BOTS. Easiest money you'll ever make. (Speaking from experience ;) )

Easy money but also unethical.

Sell something you know has a defect, going out of your way to ensure this is not obvious with the intent to sucker someone inexperienced...jikes.

Re: Ask HN: Website with 6^16 subpages and 80k+ daily bots

#57

Post by pzmarzly »

afandian wrote: post #54
PeterStuer wrote: post #51

Earlier quoted context omitted.

"If any client requests /wp-admin, flag their IP ASN as bot" You are going to hit a lot more false positives with this one than actual bots

Why? Who is legitimately going to that address but the site admin?

There are some browser plugins that try to guess what technologies are used by the website you are visiting. I hope the better ones can guess it by just looking at HTML and HTTP headers, but wouldn't be surprised if others were querying some known endpoints.

Re: Ask HN: Website with 6^16 subpages and 80k+ daily bots

#58

Post by tromp »

damir wrote: post #12
tonyg wrote: post #8

Where does the 6^16 come from? There are only 16.7 million 24-bit RGB triples; naively, if you're treating 3-hexit and 6-hexit colours separately, that'd be 16,781,312 distinct pages. What am I missing?

6 positions, each 0-F value gives 6^16 options, yes?

If you think 3 positions, each 0-1, gives 3^2 options, then please show us the 9th three-bit number. Even simpler is the case of 1 position that is 0-1. Does that give 1^2 or 2^1 options?

Re: Ask HN: Website with 6^16 subpages and 80k+ daily bots

#59

Post by discoinverno »

cookiengineer wrote: post #36

First off, make a website defend mode that can be triggered to serve different content. Then, do the following: 1. Add a robots.txt, make it look like it's wordpress (Disallow: /wp-admin etc) 2. If any client requests /wp-admin, flag their IP ASN as bot. 3. If a client is a bot, send it a gzip bomb (100kB size, unpacked around 20GB), use Transfer-Encoding: gzip and Content-Encoding: gzip to also punish malicious web…

Unrelated, but if I try to send you a message on https://cookie.engineer/contact.html it says "Could not send message, check ad-blocking extension", but I'm pretty sure I turned them off and it still doesn't work

Also, the best starter is charmender

Re: Ask HN: Website with 6^16 subpages and 80k+ daily bots

#60

Post by PeterStuer »

afandian wrote: post #54
PeterStuer wrote: post #51

Earlier quoted context omitted.

"If any client requests /wp-admin, flag their IP ASN as bot" You are going to hit a lot more false positives with this one than actual bots

Why? Who is legitimately going to that address but the site admin?

If you ban an IP or even an ASN, there could be (many) thousands sharing that same identifier. Some kid will unknowingly run some free game that does some lightweight scraping in the background as monetization and you ban the whole ISP?
Post reply on HN