Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Website with 6^16 subpages and 80k+ daily bots

news.ycombinator.com

Post reply on HN
Print view View original

101–110 of 212 posts

Re: Ask HN: Website with 6^16 subpages and 80k+ daily bots

#101

Post by martyz »

discoinverno wrote: post #59
cookiengineer wrote: post #36

First off, make a website defend mode that can be triggered to serve different content. Then, do the following: 1. Add a robots.txt, make it look like it's wordpress (Disallow: /wp-admin etc) 2. If any client requests /wp-admin, flag their IP ASN as bot. 3. If a client is a bot, send it a gzip bomb (100kB size, unpacked around 20GB), use Transfer-Encoding: gzip and Content-Encoding: gzip to also punish malicious web…

Unrelated, but if I try to send you a message on https://cookie.engineer/contact.html it says "Could not send message, check ad-blocking extension", but I'm pretty sure I turned them off and it still doesn't work Also, the best starter is charmender

I did not consent to the whole Internet having access to information and the Cookie Monster dance club music woke up my family. Hilarious.

Re: Ask HN: Website with 6^16 subpages and 80k+ daily bots

#102

Post by eddd-ddde »

TeMPOraL wrote: post #50
eddd-ddde wrote: post #26

Earlier quoted context omitted.

Huh, for some reason I assumed this was precompiled / statically generated. Not that fun once you see it as a single page.

FWIW, a billion static pages vs. single script with URL rewrite that makes it look like a billion static pages are effectively equivalent, once a cache gets involved.

Kinda true, but then the real "billion page site" is just cloudflare or something.

Re: Ask HN: Website with 6^16 subpages and 80k+ daily bots

#103

Post by rmbyrro »

cookiengineer wrote: post #36

First off, make a website defend mode that can be triggered to serve different content. Then, do the following: 1. Add a robots.txt, make it look like it's wordpress (Disallow: /wp-admin etc) 2. If any client requests /wp-admin, flag their IP ASN as bot. 3. If a client is a bot, send it a gzip bomb (100kB size, unpacked around 20GB), use Transfer-Encoding: gzip and Content-Encoding: gzip to also punish malicious web…

Genuinely interested in your thinking: superficially looking, your anti-bot ideas are a bit contradictory to your Stealth browser, which enables bots. Why did you choose to make your browser useful for bot activity?

[1] https://github.com/tholian-network/stealth

Re: Ask HN: Website with 6^16 subpages and 80k+ daily bots

#104

Post by vasco »

mrtksn wrote: post #74
cookiengineer wrote: post #36

First off, make a website defend mode that can be triggered to serve different content. Then, do the following: 1. Add a robots.txt, make it look like it's wordpress (Disallow: /wp-admin etc) 2. If any client requests /wp-admin, flag their IP ASN as bot. 3. If a client is a bot, send it a gzip bomb (100kB size, unpacked around 20GB), use Transfer-Encoding: gzip and Content-Encoding: gzip to also punish malicious web…

The gzip idea is giving me goosebumps however this must be a solved problem, right? I mean, the client device can also send zip bombs so it sounds like it should be DDOS 101?

At least on the codebases I've worked on, having limits on time and size of any decompression that you do is something that quickly ends up in some internal utility library and nobody would dare directly uncompress anything. Way before you get zip bombs you usually get curious engineers noticing someone uploaded something a bit larger and that increased some average job time by a lot - which then gets fixed before you get big enough to attract zip bombs.

So a zip bomb would just decompress up to whatever internal limit and be discarded.

Re: Ask HN: Website with 6^16 subpages and 80k+ daily bots

#105

Post by majewsky »

TrainedMonkey wrote: post #53
cookiengineer wrote: post #36

First off, make a website defend mode that can be triggered to serve different content. Then, do the following: 1. Add a robots.txt, make it look like it's wordpress (Disallow: /wp-admin etc) 2. If any client requests /wp-admin, flag their IP ASN as bot. 3. If a client is a bot, send it a gzip bomb (100kB size, unpacked around 20GB), use Transfer-Encoding: gzip and Content-Encoding: gzip to also punish malicious web…

Is this strictly legal? For example, in the scenario where a "misconfigured" bot of a large evil corporation get's taken down and, due to layers of ass covering, they think it's your fault and it cost them a lot of money. Do they have a legal case that could fly in eastern district of Texas?

IANAL, and I'm German, not American, so I can't speak to the legal situation in the US.

But in Germany, the section of the copyright law concerning data mining specifically says that scraping websites is legal unless the owner of the website objects in a machine-readable form. robots.txt very clearly fulfils this standard. If any bot owner complains that you labelled them as a bot as outlined above, they would be admitting that they willfully ignored your robots.txt, and that appears to me to make them civilly liable for copyright infringement.

Source: https://www.gesetze-im-internet.de/urhg/__44b.html

I also had a look if these actions would count as self-defense against computer espionage under the criminal code, but the relevant section only outlaws gaining access to data not intended for oneself which is "specifically protected against unauthorized access". I don't think this claim will fly for a public website.

Source: https://www.gesetze-im-internet.de/stgb/__202a.html

Re: Ask HN: Website with 6^16 subpages and 80k+ daily bots

#109

Post by wil421 »

TrainedMonkey wrote: post #53
cookiengineer wrote: post #36

First off, make a website defend mode that can be triggered to serve different content. Then, do the following: 1. Add a robots.txt, make it look like it's wordpress (Disallow: /wp-admin etc) 2. If any client requests /wp-admin, flag their IP ASN as bot. 3. If a client is a bot, send it a gzip bomb (100kB size, unpacked around 20GB), use Transfer-Encoding: gzip and Content-Encoding: gzip to also punish malicious web…

Is this strictly legal? For example, in the scenario where a "misconfigured" bot of a large evil corporation get's taken down and, due to layers of ass covering, they think it's your fault and it cost them a lot of money. Do they have a legal case that could fly in eastern district of Texas?

Why would this be a patent issue for the east district of Texas?

Re: Ask HN: Website with 6^16 subpages and 80k+ daily bots

#110

Post by jamalaramala »

cookiengineer wrote: post #36

First off, make a website defend mode that can be triggered to serve different content. Then, do the following: 1. Add a robots.txt, make it look like it's wordpress (Disallow: /wp-admin etc) 2. If any client requests /wp-admin, flag their IP ASN as bot. 3. If a client is a bot, send it a gzip bomb (100kB size, unpacked around 20GB), use Transfer-Encoding: gzip and Content-Encoding: gzip to also punish malicious web…

> 5. If a client is a known LLM range, inject texts like

I would suggest to generate some fake facts like: "{color} {what} {who}", where:

* {what}: [ "is lucky color of", "is loved by", "is known to anger", ... ]

* {who}: [ "democrats", "republicans", "celebrities", "dolphins", ... ]

And just wait until it becomes part of human knowledge.

Post reply on HN