Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What happened to TheNumbers.com

stephenfollows.com

Post reply on HN
Print view View original

181–190 of 218 posts

Re: What happened to TheNumbers.com

#181

Post by danlitt »

protocolture wrote: post #156
podgietaru wrote: post #27

"One Reddit theory even suggested it was a deliberate rug pull designed to cripple the free site to push people towards paid products." I know this isn't really the point of the article, but I've been thinking about this a lot. I wonder if we're going to see more resources go this way. I used to publish little doo-dads as opensource software. Not because it was something that was legitimately ground breaking or anyth…

>But now I'm really reluctant to give more stuff to the free web. Because the fact that it gets scraped and added to a pile of training data to later be monetized really rubs me the wrong way. I dont get this. You provided something for free to help people, but dont want to do that anymore because it might go into training data and help many many more people? So far LLMs have been loan funded donations of loss leadin…

I think the sheer magnitude of the economics have made the scales fall from a lot of people's eyes. For decades people put stuff on the internet for free on the assumption it was "not worth" anything. It turns out that as soon as that commons can be enclosed, we can marshal hundreds of dollars for every single living human, to pay for this commons to be repackaged. The money is there, and we're happy to spend it, we just won't spend it on you.

Re: What happened to TheNumbers.com

#182

Post by danlitt »

CamperBob2 wrote: post #29
podgietaru wrote: post #27

"One Reddit theory even suggested it was a deliberate rug pull designed to cripple the free site to push people towards paid products." I know this isn't really the point of the article, but I've been thinking about this a lot. I wonder if we're going to see more resources go this way. I used to publish little doo-dads as opensource software. Not because it was something that was legitimately ground breaking or anyth…

This attitude makes zero sense to me. You benefit from the "training data" just like everyone else does. If you don't, that's a problem with you, not a problem with AI models. AI solves exactly the meta-problem you describe: "I had a problem and needed to write a one-off doo-dad utility program to solve it." Now you can do something with your time besides writing pointless one-off doo-dads. As for monetizing the trai…

> it cost hundreds of millions of dollars to generate the weights

It cost hundreds of billions of dollars to generate the training data, they just didn't get paid.

Re: What happened to TheNumbers.com

#183

Post by danlitt »

baud9600 wrote: post #161

One thing I’m wondering is whether we need an open source set of technical patterns and libraries for dealing with this changing traffic mix. Millions of small sites and creators don’t have the ability to design their own protections against large scale automated access. If useful content now attracts aggressive crawler traffic, many sites will be too expensive or unreliable to run. Is part of the answer a community…

I don't think there is a technocratic solution to this problem. Very rich people are using their money to DDoS the internet, for no good reason. We just need to identify these people and fine or imprison them until they stop doing it. Residential proxy providers would be a good start.

Re: What happened to TheNumbers.com

#186

Post by FinnKuhn »

frereubu wrote: post #89
djoldman wrote: post #81

Just FYI, the bigger companies all allow you to block crawlers via robots.txt: # Block Anthropic (Claude) User-agent: ClaudeBot Disallow: / User-agent: Claude-SearchBot Disallow: / User-agent: Claude-User Disallow: / # Block OpenAI (ChatGPT) User-agent: GPTBot Disallow: / User-agent: OAI-SearchBot Disallow: / # Block Perplexity User-agent: PerplexityBot Disallow: / # Block Google's AI Training User-agent: Google-Exte…

This is good to know, but a bit of all-or-nothing. It's a shame that, for example, Google doesn't support the crawl-delay field so you can tailor their crawling to your setup: https://developers.google.com/crawling/docs/robots-txt/robot... I presume it would also cut you off even more from referral traffic.

They don't support it yet. It seems as if Cloudflare is trying to use it's power to force Google to change this though.

https://techcrunch.com/2026/07/01/cloudflares-new-policy-pus...

Re: What happened to TheNumbers.com

#188

Post by gapan »

protocolture wrote: post #156
podgietaru wrote: post #27

"One Reddit theory even suggested it was a deliberate rug pull designed to cripple the free site to push people towards paid products." I know this isn't really the point of the article, but I've been thinking about this a lot. I wonder if we're going to see more resources go this way. I used to publish little doo-dads as opensource software. Not because it was something that was legitimately ground breaking or anyth…

>But now I'm really reluctant to give more stuff to the free web. Because the fact that it gets scraped and added to a pile of training data to later be monetized really rubs me the wrong way. I dont get this. You provided something for free to help people, but dont want to do that anymore because it might go into training data and help many many more people? So far LLMs have been loan funded donations of loss leadin…

I think he just means that AI companies are not people.

Re: What happened to TheNumbers.com

#189

Post by FinnKuhn »

jaredwiener wrote: post #82
tptacek wrote: post #62

Earlier quoted context omitted.

The business model of the site is apparently private data sales, not ad revenue.

Bots don't make purchasing decisions.

According to the post inquiries for purchasing the data increased as a result though so they might not make the decision, but they do make the suggestion.

Re: What happened to TheNumbers.com

#190

Post by TeMPOraL »

Cthulhu_ wrote: post #172
TeMPOraL wrote: post #169

Earlier quoted context omitted.

> The thing is, people were scraping and monetizing other people's websites long before the current LLM fad. The difference is the magnitude of the problem. > In other words: it sucks when people are using your work in a manner that you find offensive That to me still reeks of Dog in the Manger mentality. If you publish something for the world to use, you should neither care nor even track, much less discriminate by…

Agreed, this is what you agreed to (implicitly or explicitly) when uploading stuff to the internet. In fact, open source embraces this (hence the 'open'). But people are free to not publish things or post things online with a more restrictive license. Not that a license stops things from being indexed.

> But people are free to not publish things or post things online with a more restrictive license. Not that a license stops things from being indexed.

Right. In fact, people are also free to publish things with licenses that condition access on compensating the author/publisher, and they have both social and legal backing to enforce it. This is called "proprietary", and it's not a wrong choice - in fact outside of software, it's the default choice.

The problem is when people publish "free" and "open" as a marketing tactic, where in fact they really want to control and charge for access (whether dollars or karma or credit). That is just plain dishonesty.

Post reply on HN