Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What happened to TheNumbers.com

stephenfollows.com

Post reply on HN
Print view View original

101–110 of 218 posts

Re: What happened to TheNumbers.com

#101

Post by spiderfarmer »

djoldman wrote: post #81

Just FYI, the bigger companies all allow you to block crawlers via robots.txt: # Block Anthropic (Claude) User-agent: ClaudeBot Disallow: / User-agent: Claude-SearchBot Disallow: / User-agent: Claude-User Disallow: / # Block OpenAI (ChatGPT) User-agent: GPTBot Disallow: / User-agent: OAI-SearchBot Disallow: / # Block Perplexity User-agent: PerplexityBot Disallow: / # Block Google's AI Training User-agent: Google-Exte…

90% of bot traffic on my network of websites is through headless Chrome, via residential bots nowadays. Impossible to block. Not even for Google, as they inflate my Adsense numbers as well.

Re: What happened to TheNumbers.com

#102

Post by spiderfarmer »

cogogo wrote: post #79

I think I probably use AI like a lot of consumers out there. Search engines have gotten bad and AI really good at answering fairly specific questions. Often pointing at sites like wikipedia. Pretty clear changing my behavior will have zero impact but it is certainly part of the problem. Feels a lot like my CO2 consumption. *Edit - CO2 creation

GPTBot crawled millions of pages on my website. I get a handful of visitors from them. Meanwhile Google sends me 10k visitors each day.

So GPTBot is now blocked.

Re: What happened to TheNumbers.com

#103

Post by spiderfarmer »

paxys wrote: post #31

I know people have opinions about Cloudflare but why not use it here, at least as a stop gap? Stopping bot traffic is one thing it does very well.

It tried it. Hundreds of “genuine” visitors per day on a new website with no search engine presence and no links. That’s a very leaky fence..

Re: What happened to TheNumbers.com

#105

Post by 6510 »

spiderfarmer wrote: post #101
djoldman wrote: post #81

Just FYI, the bigger companies all allow you to block crawlers via robots.txt: # Block Anthropic (Claude) User-agent: ClaudeBot Disallow: / User-agent: Claude-SearchBot Disallow: / User-agent: Claude-User Disallow: / # Block OpenAI (ChatGPT) User-agent: GPTBot Disallow: / User-agent: OAI-SearchBot Disallow: / # Block Perplexity User-agent: PerplexityBot Disallow: / # Block Google's AI Training User-agent: Google-Exte…

90% of bot traffic on my network of websites is through headless Chrome, via residential bots nowadays. Impossible to block. Not even for Google, as they inflate my Adsense numbers as well.

I once had the dumb idea to make websites in pdf but the idea is growing on me. Perhaps it should even be in animated gif with 's

Re: What happened to TheNumbers.com

#107

Post by CamperBob2 »

akudha wrote: post #84
CamperBob2 wrote: post #29

Earlier quoted context omitted.

This attitude makes zero sense to me. You benefit from the "training data" just like everyone else does. If you don't, that's a problem with you, not a problem with AI models. AI solves exactly the meta-problem you describe: "I had a problem and needed to write a one-off doo-dad utility program to solve it." Now you can do something with your time besides writing pointless one-off doo-dads. As for monetizing the trai…

It does make sense. When podgietaru wrote their doo-dad programs, I guess they made their work available for humans to use and they didn't mind giving their work away for free. Some other person might say "I'm giving my work away free, I don't care if humans use it or AI, for any purpose". Both are legitimate choices, each according to their own. As for "everyone benefiting from training data" - sure, but the AI comp…

If I stole all veggies from your garden but spent time/money/effort making a meal, I should at minimum share it with you for free, no?

Yes, and that's exactly my point. We are in violent agreement. It's shared with me, with you, with OP, and with everybody else. We can get a delicious meal for free or we can pay somebody else to serve us a slightly-tastier version.

Here, disregarding copyright law has fulfilled the very purpose of copyright law: to advance the useful arts and sciences. Copyright law was the best tool we had to accomplish that before, and now we have something better.

Re: What happened to TheNumbers.com

#108

Post by CamperBob2 »

dminik wrote: post #56
CamperBob2 wrote: post #29

Earlier quoted context omitted.

This attitude makes zero sense to me. You benefit from the "training data" just like everyone else does. If you don't, that's a problem with you, not a problem with AI models. AI solves exactly the meta-problem you describe: "I had a problem and needed to write a one-off doo-dad utility program to solve it." Now you can do something with your time besides writing pointless one-off doo-dads. As for monetizing the trai…

This is such a sad worldview. Life without any challenges. Every problem immediately solved without learning anything.

What's sad is somebody giving you a godlike tool and watching you mope around muttering about "life without any challenges."

Re: What happened to TheNumbers.com

#109

Post by theragra »

I have small website with archive of older radio broadcasts mostly in Russian. Lately, 90% of traffic comes from USA ;)

Previously it was like 5%.

URL is http://radar.lv btw

Thanks God i can serve up to a terabyte per month of traffic easily, otherwise it would be a catastrophe. I am not against bot scraping, but I worry about stability for meat visitors.

So, I think of enabling payments for website visits cloudflare recently developed.

I also have the problem of old technology like the mentioned site. While my personal blog uses static generator, archive website uses ancient Drupal version, which has no security patches for many years already.

Post reply on HN