Live data from Hacker News

Ask HN: Website with 6^16 subpages and 80k+ daily bots

news.ycombinator.com

191–200 of 212 posts

Re: Ask HN: Website with 6^16 subpages and 80k+ daily bots

#191

Earlier quoted context omitted.

It has captivated me too. At one point I realized that the set of real numbers will probably do the same (or a creative use of the set of natural numbers).

Let me know if you’re interested in working on a Library of Babel OSS implementation ever, would love to share the work with someone as fascinated by the concept as I am! waltzes_mobiles_0r@icloud.com

Found a couple for reference:

https://github.com/tdjsnelling/babel

https://github.com/cakenggt/Library-Of-Pybel

And here's an API for using the site:

https://github.com/victor-cortez/Library-of-Babel-Python-API

Then there's whatever this is:

https://github.com/DvorakDwarf/Witness-of-Babel

Re: Ask HN: Website with 6^16 subpages and 80k+ daily bots

#192
post #175

Earlier quoted context omitted.

You're probably right in regards to compression ratios, and I also think that brotli would be a much better candidate. Proxies probably won't support it as "Transfer-Encoding: br" though. > Not possible (unless you're talking double gzip). gzip's max compression ratio is 1032:1[1]. So 100kB can expand to at most ~103MB with single gzip. Not sure if I understand the rest of your argument though. If the critique is tha…

This comment would've definitely earned gold on Reddit. Here all you get is an upvote :)

I'm pretty unpopular on reddit because of my opinion about C/C++ as a language.

Most people in that ecosystem try to justify that they are somehow better when they can write pointer-magic that nobody else can understand, and feel personally attacked immediately when you mention how much less complex and more maintainable the code would have been if they would have used Go or Rust or another memory safe language.

For me, because I work in cyber, Go is kind of somewhere the middle ground between intentionally crappy C code for offensive purposes and maintainable Go code for defensive purposes. Can't use rust because you can't write exploits in Rust without nullifying the reason you used Rust in the first place :D

Go has a lot of conventions and paradigms I could have an opinion against. The point behind it is that it has opinions, at all, which makes it more maintainable, even when you don't like the enforced opinions.

I'm not saying that there is no reason to use C, there is. But every time you choose C as a language you should be aware that you chose a maintenance burden that increases the risk of the codebase to be abandoned in the future.

Re: Ask HN: Website with 6^16 subpages and 80k+ daily bots

#193

Sell a Bot IP ban-list subscription for $20/year from another host. This is what people often do with abandoned forum traffic, or hammered VoIP routers. =3

Out of curiosity, I checked and got bot hits from 20k+ unique IPs just in the last 10 days... Can easily set up bot IP lookup API...

Re: Ask HN: Website with 6^16 subpages and 80k+ daily bots

#194
post #193

Sell a Bot IP ban-list subscription for $20/year from another host. This is what people often do with abandoned forum traffic, or hammered VoIP routers. =3

Out of curiosity, I checked and got bot hits from 20k+ unique IPs just in the last 10 days... Can easily set up bot IP lookup API...

High frequency state-change propagation don't scale well on APIs. In such setups, one ends up paying for the connection setup/tear-down, and handling clients concurrency gets expensive.

Usually, a per-user access token with a 5 download limit per day is good enough, and can be scripted into peoples blacklist generation cycle.

Keep in mind, some ban-lists take awhile to compile and remove redundant subnets etc. It is important to hit the proxy/tor exit nodes first, than the usual country codes for nuisance traffic from users.

Have a nice day, =3

Re: Ask HN: Website with 6^16 subpages and 80k+ daily bots

#195

Earlier quoted context omitted.

> gzip bomb (100kB size, unpacked around 20GB) Not possible (unless you're talking double gzip). gzip's max compression ratio is 1032:1[1]. So 100kB can expand to at most ~103MB with single gzip. Brotli allows much larger compression. Here's[2] a brotli bomb I created that's 81MB compressed and 100TB uncompressed. That's a 1.2M:1 compression ratio. [1] https://stackoverflow.com/a/16794960 [2] https://github.com/googl…

You're probably right in regards to compression ratios, and I also think that brotli would be a much better candidate. Proxies probably won't support it as "Transfer-Encoding: br" though. > Not possible (unless you're talking double gzip). gzip's max compression ratio is 1032:1[1]. So 100kB can expand to at most ~103MB with single gzip. Not sure if I understand the rest of your argument though. If the critique is tha…

deflate.c appears to be doing compression. inflate.c[1] is what does decompression.

Are you saying you can modify your gzip compression code locally to generate a malformed gzip file? That wouldn't be exploiting deflate.c , that would be exploiting the receiver's decompression code, which might be inflate.c or some other implementation of gzip decompression, which might be in some other language. The language used by the compression code doesn't seem relevant to me, rather it's the language used by the decompression code that might have vulnerabilities that can be exploited. If you have a compressed gzip file that expands to more than 1032:1, the file itself is a proof of concept of the vulnerability; it doesn't matter whether the file was generated by C, Rust, Python, or in a hex editor by hand.

If you've found something in gzip code that causes it to use significantly more memory or disk space than it should (either during compression or decompression), I think that's a denial or service vulnerability and should be reported to gzip.

[1] https://git.savannah.gnu.org/cgit/gzip.git/tree/inflate.c

Re: Ask HN: Website with 6^16 subpages and 80k+ daily bots

#196

Earlier quoted context omitted.

> 5. If a client is a known LLM range, inject texts like I would suggest to generate some fake facts like: "{color} {what} {who}", where: * {what}: [ "is lucky color of", "is loved by", "is known to anger", ... ] * {who}: [ "democrats", "republicans", "celebrities", "dolphins", ... ] And just wait until it becomes part of human knowledge.

> > 5. If a client is a known LLM range, inject texts like … > I would suggest to generate some fake facts like: … Oh, I very much like this. But forget just LLM ranges, there could be many other unknown groups doing the same thing, or using residential proxy collections to forward their requests. Just add to every page a side-note of a couple of arbitrary sentences like this, with a “What Is This?” link to take conf…

I'm currently working on a project that's somewhat loosely related to what you were discussing. I'm building a webfont generator that I call "enigma-webfont", because it uses a series of rotations as a seed to "cipher" the text in the HTML in order to make it useless for LLMs, but to also to preserve it readable for humans.

The text itself without the webfont (which acts like a session, basically) is useless for any kind of machine processing, because it contains the shifted characters as UTF-8. The characters are then shifted back with a custom webfont whose seed is the same as the served HTML, but is different for each client. If you detect a non-bot user, it's currently just setting the seed/shift to 0, and serves the real plaintext, but that's optional as the user doesn't notice a difference (only maybe in the copy/paste function).

For me this was the only kind of web technology I could come up with to find a different way to serve "machine-readable" and "human-readable" content and to be able to differ between them. Anything else that's based on e.g. WebCrypto API or other code would be easily bypassed, because it can run in headless Browser instances.

Though taking screenshots in a headless chrome would kind of work to bypass this, but OCR is luckily currently kinda shitty and the development costs for something like that would explode compared to just adding another rotation mechanism in the webfont :D

Re: Ask HN: Website with 6^16 subpages and 80k+ daily bots

#197

First off, make a website defend mode that can be triggered to serve different content. Then, do the following: 1. Add a robots.txt, make it look like it's wordpress (Disallow: /wp-admin etc) 2. If any client requests /wp-admin, flag their IP ASN as bot. 3. If a client is a bot, send it a gzip bomb (100kB size, unpacked around 20GB), use Transfer-Encoding: gzip and Content-Encoding: gzip to also punish malicious web…

man you would be a good villain, wp

Re: Ask HN: Website with 6^16 subpages and 80k+ daily bots

#198

Earlier quoted context omitted.

> > 5. If a client is a known LLM range, inject texts like … > I would suggest to generate some fake facts like: … Oh, I very much like this. But forget just LLM ranges, there could be many other unknown groups doing the same thing, or using residential proxy collections to forward their requests. Just add to every page a side-note of a couple of arbitrary sentences like this, with a “What Is This?” link to take conf…

I'm currently working on a project that's somewhat loosely related to what you were discussing. I'm building a webfont generator that I call "enigma-webfont", because it uses a series of rotations as a seed to "cipher" the text in the HTML in order to make it useless for LLMs, but to also to preserve it readable for humans. The text itself without the webfont (which acts like a session, basically) is useless for any…

> If you detect a non-bot user, it's currently just setting the seed/shift to 0, and serves the real plaintext, but that's optional as the user doesn't notice a difference (only maybe in the copy/paste function).

You would probably have to keep that for accessibility purposes. Though then however you are detecting bot/not might be easily tricked by a good bot - the CAPCH arms race is currently at a point where such things exclude more human requests than automated ones…

Re: Ask HN: Website with 6^16 subpages and 80k+ daily bots

#200

Earlier quoted context omitted.

Unrelated, but if I try to send you a message on https://cookie.engineer/contact.html it says "Could not send message, check ad-blocking extension", but I'm pretty sure I turned them off and it still doesn't work Also, the best starter is charmender

> Also, the best starter is charmender Venusaur beats Charizard any time in the first edition. Given you use the correct attacks, of course, which are: Toxin and Leech Seed, Growth and Razorleaf :)

https://calc.pokemonshowdown.com/?gen=1

Charizard Fire Blast vs. Venusaur: 236-278 (65 - 76.5%) -- guaranteed 2HKO

Venusaur Razor Leaf vs. Charizard on a critical hit: 31-37 (8.6 - 10.3%) -- possibly the worst move ever

Worth noting that toxic gets downgraded to regular poison after switching out, which does 1/16th of damage every turn, and blocks other status like paralysis or sleep.

Leech seed would probably not do much, and is lost upon switching out.

Growth is ok, but since venusaur is slower, you only reduce damage by 33% on your second hit, not great.

Sleep powder would give you a chance, if it hits (75%) and if zard doesn't crit. But it would also waste your team's sleep if you are playing with sleep clause.

Q.E.D Charizard>Venusaur

That said it's not a transitive superiority, as we know Blastoise> Charizard, and Venusaur>Blastoise.

That said Charizard beats venusaur harder than venusaur beats blastoise, as the turtle has access to ice beam and blizzard, so it can at least deal non-stab super effective damage back before getting wrecked by razor leaf. And as a bonus gets a chance to crit or freeze.

Blastoise Blizzard vs. Venusaur: 158-186 (43.5 - 51.2%) -- 5.4% chance to 2HKO

Venusaur Razor Leaf vs. Blastoise on a critical hit: 256-302 (70.9 - 83.6%) -- guaranteed 2HKO

Finally while Zard doesn't have SE coverage moves against Blastois it is faster, so it can get 2 moves in before dying, either 2 slashes or a swords dance and a hyper beam, which deal the same damage but would leave zard roided against the next mon. (Or a slash +beam if you roll for a crit.)

Blastoise Surf vs. Charizard: 205-242 (57.1 - 67.4%) -- guaranteed 2HKO

+2 (swords dance) Charizard Hyper Beam vs. Blastoise: 194-228 (53.7 - 63.1%) -- guaranteed 2HKO

So while mon superiority by matchups are not transitive, that is, we cannot order starter pokemon from best to worst. We can definitely order their matchups from best to worst:

1) Zard vs Venusaur 2) Blastoise vs Zard 3) Venusaur vs Blastoise

With 2 and 3 being closer than 1 and 2.

The matchups themselves are transitive, and as such, in terms of starter matchups Charizard is the best, and Venusaur is the worst

QED

Post reply on HN