Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Website with 6^16 subpages and 80k+ daily bots

news.ycombinator.com

Post reply on HN
Print view View original

131–140 of 212 posts

Re: Ask HN: Website with 6^16 subpages and 80k+ daily bots

#131

Post by quectophoton »

2000swebgeek wrote: post #77
PeterStuer wrote: post #51

Earlier quoted context omitted.

"If any client requests /wp-admin, flag their IP ASN as bot" You are going to hit a lot more false positives with this one than actual bots

better yet, see if bots access /robots.txt, find them from there. no human looks at robots.txt :) add a captcha by limiting IP requests or return 429 to rate limit by IP. Using popular solutions like cloudflare could help reduce the load. Restrict by country. Alternatively, put in a login page which only solves the captcha and issues a session.

> no human looks at robots.txt :)

I... I do... sometimes. Mostly curiosity when the thought randomly pops on my head. I mean, I know I might be flagged by the website as someone weird/unusual/suspicious, but sometimes I do it anyway.

Btw, do you know if there's any easter egg on Hacker News' own robots.txt? Because there might be.

Re: Ask HN: Website with 6^16 subpages and 80k+ daily bots

#132

Post by aniviacat »

DeathArrow wrote: post #84
dankwizard wrote: post #34

Sell it to someone inexperienced who wants to pick up a high traffic website. Show the stats of visitors, monthly hits, etc. DO NOT MENTION BOTS. Easiest money you'll ever make. (Speaking from experience ;) )

As an inexperienced buyer I'd ask you to prove me those are real visitors and at least give me a breakup by country and region, if not also age, sex, income category. Also I would ask you to show me how much profit have you made from those visitors. I have no need for a high number of visitors if that doesn't translate into profit.

> I would ask you to show me how much profit have you made from those visitors.

Perhaps they aren't running ads, but you would. So while they make zero profit, you could make lots.

> breakup by country and region

Could easily be done for bots too. Oh look, most of my visitors are from areas with data centers, probably engineers with high income.

> if not also age, sex

That would only work if the site requires accounts

> income category

How would that ever work

Re: Ask HN: Website with 6^16 subpages and 80k+ daily bots

#134

Post by happymellon »

brazzy wrote: post #91
yjftsjthsd-h wrote: post #40

Earlier quoted context omitted.

Yes, it requires the client to try and extract the archive; https://en.wikipedia.org/wiki/Zip_bomb is the generic description.

What archive? The idea was to use Transfer-Encoding: gzip, which means the compression is a transparent part of the HTTP request which the client HTTP library will automatically try to extract.

Unless I misunderstood, there was a gzip transfer encoded gzip.

The transfer-encoding means that the proxy has to decompress a 200kb request into a 2Gb response to the client, and the client will receive a 2Gb file that will expand to 20Gb.

Small VM gets knocked offline and the proxy gets grumpy with the client for large file transfers.

Re: Ask HN: Website with 6^16 subpages and 80k+ daily bots

#135

Post by smitelli »

tonyg wrote: post #8

Where does the 6^16 come from? There are only 16.7 million 24-bit RGB triples; naively, if you're treating 3-hexit and 6-hexit colours separately, that'd be 16,781,312 distinct pages. What am I missing?

16,781,312 is the answer I got too.

One hex digit (0-F) is four bits. Six hex digits is (46) 24 bits. 24 bits is (2^24) 16.7 million combinations.

It is also legal to use three-digit colors in CSS, where (e.g.) #28d expands to #2288dd. In that context there are (43) 12 bits expressing (2^12) 4,096 colors. None of the three-digit codes from this group produce the same URL in the previous group, although all of the colors drawn on the screen are the same. For OP's purposes, these URLs are added to the others.

If one were to want to blow it up further, it's also legal to have an eight-digit hex code, where the rightmost two digits encode 256 possible levels of transparency for each color. That produces ~4 billion combinations.

Re: Ask HN: Website with 6^16 subpages and 80k+ daily bots

#136

Post by [deleted] »

rmbyrro wrote: post #124
cookiengineer wrote: post #120

Earlier quoted context omitted.

Just because web browsers like Firefox, Safari, and Chrome are trackable by default and generally care not about the anonymity of their users - doesn't mean that a web browser should behave that way. Being able to use the web anonymously is a value that should be held up against moral values. Malicious scraping of websites should be, too. Both ideas can simultaneously true, they don't have to be mutually exclusive. I…

Sure, that's why I said "superficially looking", because the back of my head said you obviously have a conscious intention consistent overall. Would you mind elaborating more on where you draw the line between "good" and "bad" behavior when it comes to scraping? Is it obeying robots.txt, the monetary purposes (profit / non-profit)?.. I personally have a hard time accepting that a website can allow Google to scrape al…

[deleted]

Re: Ask HN: Website with 6^16 subpages and 80k+ daily bots

#137

Post by quectophoton »

meindnoch wrote: post #122
mrtksn wrote: post #119

Earlier quoted context omitted.

what if you accept post for legit reasons?

Then you should use an HTTP library that lets you limit the uncompressed body size, and terminate the socket as soon as the limit is reached.

This is a good moment to remind people to always check what are the size limits and timeouts of their servers (and any other relevant limits), no matter if they are libraries or applications. Never assume the defaults are good for you, because sometimes they aren't.

For example, as much as I'd like to praise Go's stdlib because of how far it can take you, the defaults for its HTTP server are "the URL+headers of a request can take as much as 1MB" and "no timeouts at all, wait forever if necessary". See the `Server` type and the `ListenAndServe` function (not method) from the `net/http` package for more details.

They're probably trying to be as conservative as possible, but 1MB is huge.

Re: Ask HN: Website with 6^16 subpages and 80k+ daily bots

#138

Post by philsnow »

TrainedMonkey wrote: post #53
cookiengineer wrote: post #36

First off, make a website defend mode that can be triggered to serve different content. Then, do the following: 1. Add a robots.txt, make it look like it's wordpress (Disallow: /wp-admin etc) 2. If any client requests /wp-admin, flag their IP ASN as bot. 3. If a client is a bot, send it a gzip bomb (100kB size, unpacked around 20GB), use Transfer-Encoding: gzip and Content-Encoding: gzip to also punish malicious web…

Is this strictly legal? For example, in the scenario where a "misconfigured" bot of a large evil corporation get's taken down and, due to layers of ass covering, they think it's your fault and it cost them a lot of money. Do they have a legal case that could fly in eastern district of Texas?

That rocket docket is somewhat diminished after TC Heartland

https://www.bakerdonelson.com/return-of-the-rocket-docket-ne...

Re: Ask HN: Website with 6^16 subpages and 80k+ daily bots

#139

Post by figassis »

donohoe wrote: post #127
imdsm wrote: post #71

Earlier quoted context omitted.

I too wish to join this group

If this group ever convenes in NYC I will find a bar and buy the drinks just to be a fly on the wall.

Count me in, because reasons 1 and 2, and because Go.

Re: Ask HN: Website with 6^16 subpages and 80k+ daily bots

#140

Post by philsnow »

wil421 wrote: post #109
TrainedMonkey wrote: post #53

Earlier quoted context omitted.

Is this strictly legal? For example, in the scenario where a "misconfigured" bot of a large evil corporation get's taken down and, due to layers of ass covering, they think it's your fault and it cost them a lot of money. Do they have a legal case that could fly in eastern district of Texas?

Why would this be a patent issue for the east district of Texas?

Companies liked to bring patent suits there because it has historically been a very “business friendly” docket
Post reply on HN