Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Website with 6^16 subpages and 80k+ daily bots

news.ycombinator.com

Post reply on HN
Print view View original

61–70 of 212 posts

Re: Ask HN: Website with 6^16 subpages and 80k+ daily bots

#61

Post by m-i-l »

superkuh wrote: post #31

I did a $ find . -type f | wc -l in my ~/www I've been adding to for 24 years and I have somewhere around 8,476,585 files (not counting the ~250 million 30kb png tiles I have for 24/7/365 radio spectrogram zoomable maps since 2014). I get about 2-3k bot hits per day. Today's named bots: GPTBot => 726, Googlebot => 659, drive.google.com => 340, baidu => 208, Custom-AsyncHttpClient => 131, MJ12bot => 126, bingbot => 88…

Those are the good bots, which say who they are, probably respect robots.txt, and appear on various known bot lists. They are easy to deal with if you really want. But in my experience it is the bad bots you're more likely to want to deal with, and those can be very difficult, e.g. pretending to be browsers, coming from residential IP proxy farms, mutating their fingerprint too fast to appear on any known bot lists, etc.

Re: Ask HN: Website with 6^16 subpages and 80k+ daily bots

#62

Post by PeterStuer »

b112 wrote: post #55
PeterStuer wrote: post #51

Earlier quoted context omitted.

"If any client requests /wp-admin, flag their IP ASN as bot" You are going to hit a lot more false positives with this one than actual bots

Only someone poking about would ever hit that url on someone else's domain, so where's the downside? And "a lot" of false positives?? Recall, robots.txt is set to ignore this, so only malicious web scanners will hit it.

Do you own your ASN or unique IP? Do you like getting banned for the actions of others that share your ASN or IP?

Re: Ask HN: Website with 6^16 subpages and 80k+ daily bots

#63

Post by notachatbot123 »

PeterStuer wrote: post #60
afandian wrote: post #54

Earlier quoted context omitted.

Why? Who is legitimately going to that address but the site admin?

If you ban an IP or even an ASN, there could be (many) thousands sharing that same identifier. Some kid will unknowingly run some free game that does some lightweight scraping in the background as monetization and you ban the whole ISP?

> some free game that does some lightweight scraping in the background as monetization

What in the flying **. Is this a common thing?

Re: Ask HN: Website with 6^16 subpages and 80k+ daily bots

#64

Post by 123yawaworht456 »

damir wrote: post #12
tonyg wrote: post #8

Where does the 6^16 come from? There are only 16.7 million 24-bit RGB triples; naively, if you're treating 3-hexit and 6-hexit colours separately, that'd be 16,781,312 distinct pages. What am I missing?

6 positions, each 0-F value gives 6^16 options, yes?

1 byte (8 bits) is 2^8 (256 unique combinations of bits)

3 bytes (24 bits) is 2^24 (16777216 unique combinations of bits)

Re: Ask HN: Website with 6^16 subpages and 80k+ daily bots

#66

Post by carlsborg »

cookiengineer wrote: post #36

First off, make a website defend mode that can be triggered to serve different content. Then, do the following: 1. Add a robots.txt, make it look like it's wordpress (Disallow: /wp-admin etc) 2. If any client requests /wp-admin, flag their IP ASN as bot. 3. If a client is a bot, send it a gzip bomb (100kB size, unpacked around 20GB), use Transfer-Encoding: gzip and Content-Encoding: gzip to also punish malicious web…

Impressive.

Re: Ask HN: Website with 6^16 subpages and 80k+ daily bots

#67

Post by poincaredisk »

b112 wrote: post #55
PeterStuer wrote: post #51

Earlier quoted context omitted.

"If any client requests /wp-admin, flag their IP ASN as bot" You are going to hit a lot more false positives with this one than actual bots

Only someone poking about would ever hit that url on someone else's domain, so where's the downside? And "a lot" of false positives?? Recall, robots.txt is set to ignore this, so only malicious web scanners will hit it.

The downside is that you ban a whole ISP because of a single user misbehaving.

Personally I sometimes do a quick request to /wp-admin to check if a site is WordPress, so I guess that has a nonzero chance of affecting me. And when I mirror a website I almost always ignore robots.txt (I'm not a robot and I do it for myself). And when I randomly open robots.txt and see a weird url I often visit it. And these are just my quirks. Not a problem for a fun website, but please don't ban a whole IP - or even whole ISP - because of this.

Re: Ask HN: Website with 6^16 subpages and 80k+ daily bots

#68

Post by hedvig23 »

shubhamjain wrote: post #19

Unless your website has real humans visiting it, there's not a lot of value, I am afraid. The idea of many dynamically generated pages isn't new or unique. IPInfo[1] has 4B sub-pages for every IPv4 address. CompressJPEG[2] has lot of sub-pages to answer the query, "resize image to a x b". ColorHexa[3] has sub-pages for all hex colors. The easiest way to monetize is signup for AdSense and throw some ads on the page. […

There was another post on here where the creator responded, and he had intentionally built a site that had bots endlessly digging further through pages, though I can't recall which. I believe his site was pretty old too, and of simple html.

Re: Ask HN: Website with 6^16 subpages and 80k+ daily bots

#69

Post by etiennebausson »

pzmarzly wrote: post #57
afandian wrote: post #54

Earlier quoted context omitted.

Why? Who is legitimately going to that address but the site admin?

There are some browser plugins that try to guess what technologies are used by the website you are visiting. I hope the better ones can guess it by just looking at HTML and HTTP headers, but wouldn't be surprised if others were querying some known endpoints.

Then they are, by definition, bots scriping the site for informations, and should start by the robots.txt

Re: Ask HN: Website with 6^16 subpages and 80k+ daily bots

#70

Post by ColinWright »

damir wrote: post #12
tonyg wrote: post #8

Where does the 6^16 come from? There are only 16.7 million 24-bit RGB triples; naively, if you're treating 3-hexit and 6-hexit colours separately, that'd be 16,781,312 distinct pages. What am I missing?

6 positions, each 0-F value gives 6^16 options, yes?

You have 6 hex characters.

The first has 16 possible values;

The second also has 16 possible values for each of those 16, so now we have 16 times 16.

etc.

So it's a choice from 16, repeated for a total of 6 times.

That's 16 times 16 times ... times 16, which is 16^6.

Post reply on HN