Live data from Hacker News

Ask HN: Website with 6^16 subpages and 80k+ daily bots

news.ycombinator.com

121–130 of 212 posts

Re: Ask HN: Website with 6^16 subpages and 80k+ daily bots

#121

First off, make a website defend mode that can be triggered to serve different content. Then, do the following: 1. Add a robots.txt, make it look like it's wordpress (Disallow: /wp-admin etc) 2. If any client requests /wp-admin, flag their IP ASN as bot. 3. If a client is a bot, send it a gzip bomb (100kB size, unpacked around 20GB), use Transfer-Encoding: gzip and Content-Encoding: gzip to also punish malicious web…

Your website is brilliant!

Re: Ask HN: Website with 6^16 subpages and 80k+ daily bots

#122
post #119

Earlier quoted context omitted.

You close the socket as soon as you see "POST" and there's no POST handler registered.

what if you accept post for legit reasons?

Then you should use an HTTP library that lets you limit the uncompressed body size, and terminate the socket as soon as the limit is reached.

Re: Ask HN: Website with 6^16 subpages and 80k+ daily bots

#123

First off, make a website defend mode that can be triggered to serve different content. Then, do the following: 1. Add a robots.txt, make it look like it's wordpress (Disallow: /wp-admin etc) 2. If any client requests /wp-admin, flag their IP ASN as bot. 3. If a client is a bot, send it a gzip bomb (100kB size, unpacked around 20GB), use Transfer-Encoding: gzip and Content-Encoding: gzip to also punish malicious web…

Unrelated, but if I try to send you a message on https://cookie.engineer/contact.html it says "Could not send message, check ad-blocking extension", but I'm pretty sure I turned them off and it still doesn't work Also, the best starter is charmender

> Also, the best starter is charmender

Venusaur beats Charizard any time in the first edition. Given you use the correct attacks, of course, which are:

Toxin and Leech Seed, Growth and Razorleaf :)

Re: Ask HN: Website with 6^16 subpages and 80k+ daily bots

#124

Earlier quoted context omitted.

Genuinely interested in your thinking: superficially looking, your anti-bot ideas are a bit contradictory to your Stealth browser, which enables bots. Why did you choose to make your browser useful for bot activity? [1] https://github.com/tholian-network/stealth

Just because web browsers like Firefox, Safari, and Chrome are trackable by default and generally care not about the anonymity of their users - doesn't mean that a web browser should behave that way. Being able to use the web anonymously is a value that should be held up against moral values. Malicious scraping of websites should be, too. Both ideas can simultaneously true, they don't have to be mutually exclusive. I…

Sure, that's why I said "superficially looking", because the back of my head said you obviously have a conscious intention consistent overall.

Would you mind elaborating more on where you draw the line between "good" and "bad" behavior when it comes to scraping?

Is it obeying robots.txt, the monetary purposes (profit / non-profit)?..

I personally have a hard time accepting that a website can allow Google to scrape all their content, but prevent others. In my view, this is anti-competitive behavior favoring a large and evil corporation. Argumenting that Google delivers value back in the form of traffic is unacceptable to me, because it goes against basic principles of net neutrality, which I support.

I'm intetested in reading your thoughts on this.

Re: Ask HN: Website with 6^16 subpages and 80k+ daily bots

#126

Earlier quoted context omitted.

> some free game that does some lightweight scraping in the background as monetization What in the flying **. Is this a common thing?

For some definition of "common", yes. Some try to be less shady by asking for consent (e.g. in exchange for in-game credits), others are essentially malware. For example: https://bright-sdk.com/ > Bright SDK is approved by Apple, Amazon, LG, Huawei, Samsung app stores, and is whitelisted by top Antivirus companies.

the idea that games should be written solely to extract revenue from players is so repulsive to me that I actively disrespect and disfavor people I know who work on things like this.

humans are a truly horrible species and this kind of thing is a great example of why I believe that.

Re: Ask HN: Website with 6^16 subpages and 80k+ daily bots

#127
post #71
post #44

Earlier quoted context omitted.

I would like to be your friend for 2 reasons. #1 is that you’re brilliantly devious. #2 is that I fervently wish to stay on your good side.

I too wish to join this group

If this group ever convenes in NYC I will find a bar and buy the drinks just to be a fly on the wall.

Re: Ask HN: Website with 6^16 subpages and 80k+ daily bots

#128

First off, make a website defend mode that can be triggered to serve different content. Then, do the following: 1. Add a robots.txt, make it look like it's wordpress (Disallow: /wp-admin etc) 2. If any client requests /wp-admin, flag their IP ASN as bot. 3. If a client is a bot, send it a gzip bomb (100kB size, unpacked around 20GB), use Transfer-Encoding: gzip and Content-Encoding: gzip to also punish malicious web…

> 5. If a client is a known LLM range, inject texts like I would suggest to generate some fake facts like: "{color} {what} {who}", where: * {what}: [ "is lucky color of", "is loved by", "is known to anger", ... ] * {who}: [ "democrats", "republicans", "celebrities", "dolphins", ... ] And just wait until it becomes part of human knowledge.

Make it all clickbait titles, too.

"You won't believe what celebrities love {color}!!"

Re: Ask HN: Website with 6^16 subpages and 80k+ daily bots

#129

First off, make a website defend mode that can be triggered to serve different content. Then, do the following: 1. Add a robots.txt, make it look like it's wordpress (Disallow: /wp-admin etc) 2. If any client requests /wp-admin, flag their IP ASN as bot. 3. If a client is a bot, send it a gzip bomb (100kB size, unpacked around 20GB), use Transfer-Encoding: gzip and Content-Encoding: gzip to also punish malicious web…

That was beautiful to read lol.

Re: Ask HN: Website with 6^16 subpages and 80k+ daily bots

#130

Earlier quoted context omitted.

"If any client requests /wp-admin, flag their IP ASN as bot" You are going to hit a lot more false positives with this one than actual bots

better yet, see if bots access /robots.txt, find them from there. no human looks at robots.txt :) add a captcha by limiting IP requests or return 429 to rate limit by IP. Using popular solutions like cloudflare could help reduce the load. Restrict by country. Alternatively, put in a login page which only solves the captcha and issues a session.

> no human looks at robots.txt

of course people look at this. it's not an everyday thing for the prototypical web user, but some of us look at those a lot.

Post reply on HN