Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What happened to TheNumbers.com

stephenfollows.com

Post reply on HN
Print view View original

121–130 of 218 posts

Re: What happened to TheNumbers.com

#121

Post by philipkglass »

paxys wrote: post #31

I know people have opinions about Cloudflare but why not use it here, at least as a stop gap? Stopping bot traffic is one thing it does very well.

I set Cloudflare up a couple of months ago specifically to block bot traffic. It didn't do anything for me. Dumb bots were still hitting every special link on my wiki fast enough that the server was continually swamped running Lua scripts. 65% of the traffic for my English-language site was coming from Vietnam. But I didn't want to block Vietnam altogether, because my hobby site has genuine users from there too.

I eventually just shut down my mediawiki instance. I couldn't find a way to keep it online and still run on an affordable VPS.

Re: What happened to TheNumbers.com

#122

Post by tekne »

pavel_lishin wrote: post #67
dpoloncsak wrote: post #64

Earlier quoted context omitted.

The crawlers would have still just hammered their site though, right?

I think the idea is that they could store the data in BigQuery, and point users of the site there.

There is also the deep magic...

https://github.com/phiresky/sql.js-httpvfs

Re: What happened to TheNumbers.com

#123

Post by primitivesuave »

x3haloed wrote: post #94
primitivesuave wrote: post #25

A couple years ago, I was running a website which allowed the public to view all the US government handouts to small businesses during the COVID-19 pandemic. It also tracked all the fraudulent loans being prosecuted by the DOJ, and allowed anyone to run structured queries over the public dataset. There was a "donate" button which took in ~$2k in donations over the lifetime of the site, and you could download the enti…

That’s a traffic design problem. You should be happy that your work is valuable and also protected it against excessive requests. Simple.

I deployed a Lambda function behind CloudFront which rendered a simple HTML page with the query results from executing some SQL over the dataset. I served millions of page views for next to nothing because most pages were already in the cache.

I don't know where you get this expectation that people should anticipate that a crawler might try every possible combination of query parameters, thereby missing the cache on each one. Most people consider it a bitter and arrogant perspective, which is why this got downvoted.

Re: What happened to TheNumbers.com

#124

Post by pixelesque »

djoldman wrote: post #81

Just FYI, the bigger companies all allow you to block crawlers via robots.txt: # Block Anthropic (Claude) User-agent: ClaudeBot Disallow: / User-agent: Claude-SearchBot Disallow: / User-agent: Claude-User Disallow: / # Block OpenAI (ChatGPT) User-agent: GPTBot Disallow: / User-agent: OAI-SearchBot Disallow: / # Block Perplexity User-agent: PerplexityBot Disallow: / # Block Google's AI Training User-agent: Google-Exte…

Claude Bot still (at least last month, and it's been doing it for over a year now) seems to have a bug when traversing (at least my sites), wherein it drops the trailing slash of a directory (which is present in the a href tag), then makes the request to the subdirectory without the slash I put in the link, then Caddy automatically responds to that (via the built-in File handler) with a redirect telling it to add the trailing slash, and Claude Bot then makes the request again with the trailing slash that should have been there in the first place.

So most subdirectory URLs get two requests from Claude bot, the first one needless because that wasn't the URL in the tag.

Re: What happened to TheNumbers.com

#125

Post by ropable »

This is another reason we can't have nice things. The level of entitlement required to send an angry message to someone complaining about their free resource being offline is mind-boggling, though.

Re: What happened to TheNumbers.com

#126

Post by wredcoll »

polio wrote: post #74
ajkjk wrote: post #71

It feels like there is fundamentally missing infrastructure here that is needed to make these problems go away. Basically bots need to be (somehow) paying for the traffic they create, or prevented from creating it, or told to go away and then fined if they violate the request. No idea how to do these or even at what level in the stack they should happen, but they need to happen eventually somehow.

Every website I visit could get a fraction of a cent in my Cloudflare Wallet. A human browsing incurs a few dollars a month. Plus, any website I access in this way decides not to show me ads either and just charges me the cost of serving the data plus a nominal profit margin. Disclosure: I am a shareholder and would love for them to solve the AI bot problem and the ad problem like this.

Maybe we'll all eat our words and cryptocurrencies will actually become useful for something.

Re: What happened to TheNumbers.com

#127

Post by foco_tubi »

CamperBob2 wrote: post #108
dminik wrote: post #56

Earlier quoted context omitted.

This is such a sad worldview. Life without any challenges. Every problem immediately solved without learning anything.

What's sad is somebody giving you a godlike tool and watching you mope around muttering about "life without any challenges."

The godlike tools are paid subscriptions at best and restricted to a handful of corpos by the government at best. The free scraps thrown to the proles are mostly novelty. The other day I asked 5.6 Sol to identify a push prop airplane and it said “this is a helicopter.”

Re: What happened to TheNumbers.com

#128

Post by atherton94027 »

zackmorris wrote: post #42
ethagnawl wrote: post #4

At the risk of oversimplifying things from a distance, this site -- especially the free, public-facing part of it -- seems like it would be an ideal candidate for a rewrite using static site generator/framework. That, coupled with a bot-aware CDN should keep them online at a reasonable cost for many years to come. Otherwise, I'm very curious to know more about their old and new architecture and what sorts of mitigati…

Ya this problem was solved 20 years ago with Varnish cache and Coral cache/CDN. I think what's really going on is that bots expose how underpowered web servers has gotten in recent years. In the 2000s, even poorly-architected PHP sites tended to serve about 200 requests per second, with 1000+ being common for static sites. I remember when Node.js came out and claimed that it could serve more like 100,000 RPS due to i…

> I think what's really going on is that bots expose how underpowered web servers has gotten in recent years

More often than not they're running on a VPS, and cloud providers have been pushing the envelope of what a vCPU is. Amazon still bills hyperthreads as a single core!

Add to that their servers are often aging and you get a recipe for slow web services

Re: What happened to TheNumbers.com

#130

Post by weird-eye-issue »

wredcoll wrote: post #126
polio wrote: post #74

Earlier quoted context omitted.

Every website I visit could get a fraction of a cent in my Cloudflare Wallet. A human browsing incurs a few dollars a month. Plus, any website I access in this way decides not to show me ads either and just charges me the cost of serving the data plus a nominal profit margin. Disclosure: I am a shareholder and would love for them to solve the AI bot problem and the ad problem like this.

Maybe we'll all eat our words and cryptocurrencies will actually become useful for something.

This does not require cryptocurrency
Post reply on HN