Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What happened to TheNumbers.com

stephenfollows.com

Post reply on HN
Print view View original

81–90 of 218 posts

Re: What happened to TheNumbers.com

#81

Post by djoldman »

Just FYI, the bigger companies all allow you to block crawlers via robots.txt:

    # Block Anthropic (Claude)
    User-agent: ClaudeBot
    Disallow: /
    User-agent: Claude-SearchBot
    Disallow: /
    User-agent: Claude-User
    Disallow: /

    # Block OpenAI (ChatGPT)
    User-agent: GPTBot
    Disallow: /
    User-agent: OAI-SearchBot
    Disallow: /

    # Block Perplexity
    User-agent: PerplexityBot
    Disallow: /

    # Block Google's AI Training
    User-agent: Google-Extended
    Disallow: /
    User-agent: Google-Extended-Factual
    Disallow: /

    # Block Microsoft's Search & AI Crawler
    User-agent: Bingbot
    Disallow: /

Re: What happened to TheNumbers.com

#82

Post by jaredwiener »

tptacek wrote: post #62
jaredwiener wrote: post #17

Earlier quoted context omitted.

But it also destroys the business model behind the site. Sure, you could technically redesign to handle the bot traffic, but if the bot traffic is just taking the data and reducing any need for humans to visit the site, why is he putting in the effort to maintain the site?

The business model of the site is apparently private data sales, not ad revenue.

Bots don't make purchasing decisions.

Re: What happened to TheNumbers.com

#83

Post by lokar »

deepsun wrote: post #48
primitivesuave wrote: post #25

A couple years ago, I was running a website which allowed the public to view all the US government handouts to small businesses during the COVID-19 pandemic. It also tracked all the fraudulent loans being prosecuted by the DOJ, and allowed anyone to run structured queries over the public dataset. There was a "donate" button which took in ~$2k in donations over the lifetime of the site, and you could download the enti…

BigQuery has "public datasets", so users can even run complex SQL on it, but it's them who pays for it, not you. You only pay for data storage.

I think snowflake has similar, you can rent it out or make it free

Re: What happened to TheNumbers.com

#84

Post by akudha »

CamperBob2 wrote: post #29
podgietaru wrote: post #27

"One Reddit theory even suggested it was a deliberate rug pull designed to cripple the free site to push people towards paid products." I know this isn't really the point of the article, but I've been thinking about this a lot. I wonder if we're going to see more resources go this way. I used to publish little doo-dads as opensource software. Not because it was something that was legitimately ground breaking or anyth…

This attitude makes zero sense to me. You benefit from the "training data" just like everyone else does. If you don't, that's a problem with you, not a problem with AI models. AI solves exactly the meta-problem you describe: "I had a problem and needed to write a one-off doo-dad utility program to solve it." Now you can do something with your time besides writing pointless one-off doo-dads. As for monetizing the trai…

It does make sense. When podgietaru wrote their doo-dad programs, I guess they made their work available for humans to use and they didn't mind giving their work away for free. Some other person might say "I'm giving my work away free, I don't care if humans use it or AI, for any purpose". Both are legitimate choices, each according to their own.

As for "everyone benefiting from training data" - sure, but the AI companies are not investing Billions of dollars out of the goodness of their hearts, it is towards one and only singular goal of making profits (at some point). People might be sympathetic to these AI companies if they at least behave decently - they take everyone's work (text, software, fiction, music, images, videos...) without paying a penny to anyone. If they take everyone's work for free, they should give away anything that is built on that work also for free. This is before we even get to environment, privacy, hammering sites by not respecting robots.txt etc issues.

If I stole all veggies from your garden but spent time/money/effort making a meal, I should at minimum share it with you for free, no? Even if I spent my own money making the meal, it was made from stolen raw material...

Re: What happened to TheNumbers.com

#85

Post by 40four »

PunchyHamster wrote: post #69
pavel_lishin wrote: post #67

Earlier quoted context omitted.

I think the idea is that they could store the data in BigQuery, and point users of the site there.

The crawlers would have still just hammered their site though, right?

Why is your comment exactly word for word of another comment just one level above in the comment chain?

Re: What happened to TheNumbers.com

#86

Post by taneq »

40four wrote: post #85
PunchyHamster wrote: post #69

Earlier quoted context omitted.

The crawlers would have still just hammered their site though, right?

Why is your comment exactly word for word of another comment just one level above in the comment chain?

Because the crawlers would still have hammered their site, though.

(The GP post doesn’t actually meaningfully address the issue being raised. Adding BigQuery or whatever would not change the fact that (a) they already offered a method of getting all of the data in a cost effective way, and (b) the issue was that the crawlers hammered the site hard enough to make it economically unviable.)

Re: What happened to TheNumbers.com

#87

Post by pavel_lishin »

dpoloncsak wrote: post #75
pavel_lishin wrote: post #67

Earlier quoted context omitted.

I think the idea is that they could store the data in BigQuery, and point users of the site there.

Sure, but now you're moving the site from "Free data presented in a pleasant way to view" to a "pay-as-you-go database". Your audience shifts dramatically, and you lose the ability to share the data you're trying to present.

True. But it sounds like they already lost that.

Re: What happened to TheNumbers.com

#88

Post by gorgonian »

40four wrote: post #85
PunchyHamster wrote: post #69

Earlier quoted context omitted.

The crawlers would have still just hammered their site though, right?

Why is your comment exactly word for word of another comment just one level above in the comment chain?

Maybe because they restated what they said instead of addressing to the previous commenter’s point.

Re: What happened to TheNumbers.com

#89

Post by frereubu »

djoldman wrote: post #81

Just FYI, the bigger companies all allow you to block crawlers via robots.txt: # Block Anthropic (Claude) User-agent: ClaudeBot Disallow: / User-agent: Claude-SearchBot Disallow: / User-agent: Claude-User Disallow: / # Block OpenAI (ChatGPT) User-agent: GPTBot Disallow: / User-agent: OAI-SearchBot Disallow: / # Block Perplexity User-agent: PerplexityBot Disallow: / # Block Google's AI Training User-agent: Google-Exte…

This is good to know, but a bit of all-or-nothing. It's a shame that, for example, Google doesn't support the crawl-delay field so you can tailor their crawling to your setup: https://developers.google.com/crawling/docs/robots-txt/robot...

I presume it would also cut you off even more from referral traffic.

Post reply on HN