Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What happened to TheNumbers.com

stephenfollows.com

Post reply on HN
Print view View original

161–170 of 218 posts

Re: What happened to TheNumbers.com

#161

Post by baud9600 »

One thing I’m wondering is whether we need an open source set of technical patterns and libraries for dealing with this changing traffic mix.

Millions of small sites and creators don’t have the ability to design their own protections against large scale automated access. If useful content now attracts aggressive crawler traffic, many sites will be too expensive or unreliable to run.

Is part of the answer a community response? Perhaps a community-maintained toolkit, based on traffic data, that host sites can apply? It could include standard agent identification, rate limiting, traffic classification, access policies, caching, challenge mechanisms, logging, attribution and usage control etc etc.

In effect, we need much stronger road rules for today’s automated traffic, available as open technical patterns and libraries rather than every site owner having to invent this alone (they won’t).

Re: What happened to TheNumbers.com

#162

Post by dotancohen »

joshmarinacci wrote: post #46
II2II wrote: post #45

Earlier quoted context omitted.

> But now I'm really reluctant to give more stuff to the free web. Because the fact that it gets scraped and added to a pile of training data to later be monetized really rubs me the wrong way. The thing is, people were scraping and monetizing other people's websites long before the current LLM fad. The difference is the magnitude of the problem. > And I can see why maintainers of sites like this, or other free but i…

A change in quantity can become a change in quality.

I believe that it was Stalin who phrased it most eloquently: Quantity has a quality all its own.

Re: What happened to TheNumbers.com

#163

Post by dotancohen »

ryukoposting wrote: post #133
podgietaru wrote: post #27

"One Reddit theory even suggested it was a deliberate rug pull designed to cripple the free site to push people towards paid products." I know this isn't really the point of the article, but I've been thinking about this a lot. I wonder if we're going to see more resources go this way. I used to publish little doo-dads as opensource software. Not because it was something that was legitimately ground breaking or anyth…

> Because the fact that it gets scraped and added to a pile of training data to later be monetized really rubs me the wrong way. I suspect the next AGPL (if not normal GPL) will explicitly prohibit training of closed models against the source material. Ideally it will prohibit training of open weight models too. Either share the entire process, or go piss up a rope.

My understanding of the GPL is that it currently prohibits the training of LLMs, no need to explicitly mention them. MIT does allow it, however.

Re: What happened to TheNumbers.com

#164

Post by dotancohen »

bigstrat2003 wrote: post #77
vachina wrote: post #57

Earlier quoted context omitted.

> But now I'm really reluctant to give more stuff to the free web. I feel the same way too. But guess what, all the code you did not publish gets into the training corpus anyway (when you gave Codex or whatever full read access to your filesystem).

That's only if one is stupid enough to give an LLM access to their computer. Don't do that, it's incredibly irresponsible security wise.

The vast majority of people are in fact this stupid.

I recently had trouble connecting to a new AP on my laptop. After a quarter hour of frustration, I connected to the AP of my phone, asked Claude Code what the problem is, and she found the issue in seconds. I didn't allow her to make the actual changes, but she did have read access to everything, and helped me considerably.

So network-manager gets a new bug report about too-long non-ASCII AP names, I get online, and I don't know maybe Anthropic sneakily learned something from my local python projects. I am one of those vast-majority stupid people.

Re: What happened to TheNumbers.com

#166

Post by primitivesuave »

tekne wrote: post #122
pavel_lishin wrote: post #67

Earlier quoted context omitted.

I think the idea is that they could store the data in BigQuery, and point users of the site there.

There is also the deep magic... https://github.com/phiresky/sql.js-httpvfs

I'm curious how this compares to just using DuckDB in the browser?

https://duckdb.org/2021/10/29/duckdb-wasm

Re: What happened to TheNumbers.com

#167

Post by globular-toast »

djoldman wrote: post #81

Just FYI, the bigger companies all allow you to block crawlers via robots.txt: # Block Anthropic (Claude) User-agent: ClaudeBot Disallow: / User-agent: Claude-SearchBot Disallow: / User-agent: Claude-User Disallow: / # Block OpenAI (ChatGPT) User-agent: GPTBot Disallow: / User-agent: OAI-SearchBot Disallow: / # Block Perplexity User-agent: PerplexityBot Disallow: / # Block Google's AI Training User-agent: Google-Exte…

I won't eat your lunch if you put a sticker on your lunch box telling me not to.

Re: What happened to TheNumbers.com

#169

Post by TeMPOraL »

II2II wrote: post #45
podgietaru wrote: post #27

"One Reddit theory even suggested it was a deliberate rug pull designed to cripple the free site to push people towards paid products." I know this isn't really the point of the article, but I've been thinking about this a lot. I wonder if we're going to see more resources go this way. I used to publish little doo-dads as opensource software. Not because it was something that was legitimately ground breaking or anyth…

> But now I'm really reluctant to give more stuff to the free web. Because the fact that it gets scraped and added to a pile of training data to later be monetized really rubs me the wrong way. The thing is, people were scraping and monetizing other people's websites long before the current LLM fad. The difference is the magnitude of the problem. > And I can see why maintainers of sites like this, or other free but i…

> The thing is, people were scraping and monetizing other people's websites long before the current LLM fad. The difference is the magnitude of the problem.

> In other words: it sucks when people are using your work in a manner that you find offensive

That to me still reeks of Dog in the Manger mentality. If you publish something for the world to use, you should neither care nor even track, much less discriminate by (or suddenly seek compensation for) who is using it.

Re: What happened to TheNumbers.com

#170

Post by Cthulhu_ »

baud9600 wrote: post #161

One thing I’m wondering is whether we need an open source set of technical patterns and libraries for dealing with this changing traffic mix. Millions of small sites and creators don’t have the ability to design their own protections against large scale automated access. If useful content now attracts aggressive crawler traffic, many sites will be too expensive or unreliable to run. Is part of the answer a community…

The community response was respecting robots.txt, but since that was more of a gentleman's agreement than a legal requirement, hungry AI scrapers disregarded it.

Which brings us to the old fashioned flood control mechanisms. That is, the toolkit you propose already exists and has been used for decades in various iterations to protect against various forms of attack (slashdotting, DDOS attacks, overzealous search engines, and now AI scrapers).

Have you looked into those before? Companies like Cloudflare have been at the forefront of this field for a long time now.

Post reply on HN