Earlier quoted context omitted.
Building on AWS is a financial time bomb.
Are you not able to put limits on how much the site can spend?
As of 2026, still not, and probably never.
What happened to TheNumbers.com
91–100 of 218 posts
Re: What happened to TheNumbers.com
#92At the risk of oversimplifying things from a distance, this site -- especially the free, public-facing part of it -- seems like it would be an ideal candidate for a rewrite using static site generator/framework. That, coupled with a bot-aware CDN should keep them online at a reasonable cost for many years to come. Otherwise, I'm very curious to know more about their old and new architecture and what sorts of mitigati…
Ya this problem was solved 20 years ago with Varnish cache and Coral cache/CDN. I think what's really going on is that bots expose how underpowered web servers has gotten in recent years. In the 2000s, even poorly-architected PHP sites tended to serve about 200 requests per second, with 1000+ being common for static sites. I remember when Node.js came out and claimed that it could serve more like 100,000 RPS due to i…
> Nearly everything should be available from a nearby edge peer, similarly to BitTorrent.
Back in the days of Gnutella, I remember pushing people to use Magnet links [0] when sharing content.
Re: What happened to TheNumbers.com
#93Re: What happened to TheNumbers.com
#94A couple years ago, I was running a website which allowed the public to view all the US government handouts to small businesses during the COVID-19 pandemic. It also tracked all the fraudulent loans being prosecuted by the DOJ, and allowed anyone to run structured queries over the public dataset. There was a "donate" button which took in ~$2k in donations over the lifetime of the site, and you could download the enti…
That’s a traffic design problem. You should be happy that your work is valuable and also protected it against excessive requests. Simple.
Re: What happened to TheNumbers.com
#95All new content will be behind paywalls. This will be the new way the Internet works and I've said this for the last several years. There's no value in writing unique content just to have AI steal it and distribute it without you getting any clicks. The only way it works if it there's a licensing agreement and if they don't want to pay, then who cares, you weren't going to make any money from them anyway.
Re: What happened to TheNumbers.com
#96Ironic that polymarkets were being advertised as helping society make better predictions.
I mean wisdom of crowds, super-forecasters, calibration and pre-registration are useful tools that can result in better predictions, turning it into online gambling is where it went sideways.
Mini-rant: Promoters claim that the system serves a public good by helping society discover/converge on useful truths sooner.
Even where the question/event is of public interest, the opposite happens instead. People with expertise or non-public information are incentivized to misdirect and delay as long as possible, as that maximizes what they can make from betting.
Re: What happened to TheNumbers.com
#97All new content will be behind paywalls. This will be the new way the Internet works and I've said this for the last several years. There's no value in writing unique content just to have AI steal it and distribute it without you getting any clicks. The only way it works if it there's a licensing agreement and if they don't want to pay, then who cares, you weren't going to make any money from them anyway.
What if you aren't motivated solely by making money? There will still be new free content created by hobbyists, passionate creators, altruists, etc.
(Agree with you more generally.)
Re: What happened to TheNumbers.com
#98Re: What happened to TheNumbers.com
#99Earlier quoted context omitted.
Who are running these bots? I presume developers at all of the frontier labs know (or at least would know to look for) Wikipedia has bulk APIs for automated access. Unnecessary scraping increases their workload/costs too, so why in 2026 is this still a problem?
Black market and gray market data. All the firms want data. All the other firms want data. The banks want data. The other criminals also want data for their crimes and schemes. Oh insurance companies, and the ATS systems. Everybody wants as much data as they can get and they don't care how they get it.
This data selling also happens with leaks of all kinds like medical data, often to current or future employers, health insurance companies, etc.
Re: What happened to TheNumbers.com
#100All new content will be behind paywalls. This will be the new way the Internet works and I've said this for the last several years. There's no value in writing unique content just to have AI steal it and distribute it without you getting any clicks. The only way it works if it there's a licensing agreement and if they don't want to pay, then who cares, you weren't going to make any money from them anyway.
What if you aren't motivated solely by making money? There will still be new free content created by hobbyists, passionate creators, altruists, etc. (Agree with you more generally.)
Why would you produce content and have no one read it or visits your site, but OpenAI and Anthropic make millions from it? At some point it becomes stupid to just give free money to these companies when they steal literally all your traffic and content. As per the article, Anthropic sends 1 page view for every 38,000 views they get.