Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What happened to TheNumbers.com

stephenfollows.com

Post reply on HN
Print view View original

111–120 of 218 posts

Re: What happened to TheNumbers.com

#111

Post by Scoundreller »

da02 wrote: post #73
Scoundreller wrote: post #55

Earlier quoted context omitted.

I feel the same about my old blog articles. I used to rank pretty well trashing crappy credit cards and encouraging people to switch to better options, then Google decided that 10 results for the card issuers website was better. Same shit when I manually wrote proto-gethuman posts on calling telecoms/banks/etc (and also pushed visitors to try an Indy ISP or credit unions), then Google felt it was better to drive user…

Do you have any active sites or social media with your recommendations? (I never publish my recommendations because they seem to complicated for people. Like using 30 GB plan/$10/monthly from T-Mobile for data and then using Tello for $10 plan for voice/text. This would require a 2 esim/sim card phone. I am currently using a Moto G Power 2024 phone from eBay $90/new, which was better than the $200 slightly used Pixel…

Nothing active anymore, all converted to static sites.

If you’re on page 2 of the results, you effectively don’t exist so I stopped bothering/benefiting from display ads.

Coincidentally, I did try to get deeper into the cellular service side (it’s another high margin and high customer value segment), but I did better on the finance side.

Re: What happened to TheNumbers.com

#112

Post by paxys »

spiderfarmer wrote: post #103
paxys wrote: post #31

I know people have opinions about Cloudflare but why not use it here, at least as a stop gap? Stopping bot traffic is one thing it does very well.

It tried it. Hundreds of “genuine” visitors per day on a new website with no search engine presence and no links. That’s a very leaky fence..

Hundreds is better than billions.

Re: What happened to TheNumbers.com

#113

Post by ssl-3 »

freediddy wrote: post #100
npilk wrote: post #97

Earlier quoted context omitted.

What if you aren't motivated solely by making money? There will still be new free content created by hobbyists, passionate creators, altruists, etc. (Agree with you more generally.)

Why would you produce content and have no one read it or visits your site, but OpenAI and Anthropic make millions from it? At some point it becomes stupid to just give free money to these companies when they steal literally all your traffic and content. As per the article, Anthropic sends 1 page view for every 38,000 views they get.

Very few people showed up to read stuff on the old web, and yet: It existed.

Re: What happened to TheNumbers.com

#114

Post by primitivesuave »

BeeOnRope wrote: post #49
primitivesuave wrote: post #25

A couple years ago, I was running a website which allowed the public to view all the US government handouts to small businesses during the COVID-19 pandemic. It also tracked all the fraudulent loans being prosecuted by the DOJ, and allowed anyone to run structured queries over the public dataset. There was a "donate" button which took in ~$2k in donations over the lifetime of the site, and you could download the enti…

Do you have any view on why the AI scrapers resulted in a heavier load than existing crawlers from eg search engines? Where they more exhaustive or more frequent?

They were both more exhaustive and more frequent. I don't remember the exact numbers, but it was definitely over 100x the traffic from search engines. By the way, search engines were allowed under robots.txt - I did want all 11.5 million loans to be individually indexed so they would pop up in Google search results, and I actually did receive/forward multiple tips about fraudulent loans because someone searched a business name on Google. All of this traffic was barely a blip, and my AWS bill for my hobby data science account was only ~$50-$100/month.

When I looked at the logs after getting the billing alert, 99.99% of the requests were to the "/search" endpoint with virtually every permutation of ~10-12 facets in the query parameters. There was only one scraper, but it triggered an enormous amount of network egress since it ended up missing the cache on the majority of queries.

Re: What happened to TheNumbers.com

#115

Post by cycomanic »

CamperBob2 wrote: post #107
akudha wrote: post #84

Earlier quoted context omitted.

It does make sense. When podgietaru wrote their doo-dad programs, I guess they made their work available for humans to use and they didn't mind giving their work away for free. Some other person might say "I'm giving my work away free, I don't care if humans use it or AI, for any purpose". Both are legitimate choices, each according to their own. As for "everyone benefiting from training data" - sure, but the AI comp…

If I stole all veggies from your garden but spent time/money/effort making a meal, I should at minimum share it with you for free, no? Yes, and that's exactly my point. We are in violent agreement. It's shared with me, with you, with OP, and with everybody else. We can get a delicious meal for free or we can pay somebody else to serve us a slightly-tastier version. Here, disregarding copyright law has fulfilled the v…

But the AI companies are not publishing the models? Moreover they are charging for access to the models.

Re: What happened to TheNumbers.com

#116

Post by podgietaru »

CamperBob2 wrote: post #107
akudha wrote: post #84

Earlier quoted context omitted.

It does make sense. When podgietaru wrote their doo-dad programs, I guess they made their work available for humans to use and they didn't mind giving their work away for free. Some other person might say "I'm giving my work away free, I don't care if humans use it or AI, for any purpose". Both are legitimate choices, each according to their own. As for "everyone benefiting from training data" - sure, but the AI comp…

If I stole all veggies from your garden but spent time/money/effort making a meal, I should at minimum share it with you for free, no? Yes, and that's exactly my point. We are in violent agreement. It's shared with me, with you, with OP, and with everybody else. We can get a delicious meal for free or we can pay somebody else to serve us a slightly-tastier version. Here, disregarding copyright law has fulfilled the v…

I wouldn’t be happy if those companies used my vegetables to open a Michelin star restaurant, even if I got a free McDouble out of the deal.

Plant based I guess I don’t know metaphors are hard.

Re: What happened to TheNumbers.com

#117

Post by primitivesuave »

tailscaler2026 wrote: post #41
primitivesuave wrote: post #25

A couple years ago, I was running a website which allowed the public to view all the US government handouts to small businesses during the COVID-19 pandemic. It also tracked all the fraudulent loans being prosecuted by the DOJ, and allowed anyone to run structured queries over the public dataset. There was a "donate" button which took in ~$2k in donations over the lifetime of the site, and you could download the enti…

Building on AWS is a financial time bomb.

Completely agree, I've worked at several companies where I saw the AWS bill balloon from five to six figures, usually because of pointless over-provisioning. However, it all became worth it when I saw Jeff Bezos go to space. [1]

1. https://www.youtube.com/watch?v=IOmX793-5t4

Re: What happened to TheNumbers.com

#118

Post by CamperBob2 »

cycomanic wrote: post #115
CamperBob2 wrote: post #107

Earlier quoted context omitted.

If I stole all veggies from your garden but spent time/money/effort making a meal, I should at minimum share it with you for free, no? Yes, and that's exactly my point. We are in violent agreement. It's shared with me, with you, with OP, and with everybody else. We can get a delicious meal for free or we can pay somebody else to serve us a slightly-tastier version. Here, disregarding copyright law has fulfilled the v…

But the AI companies are not publishing the models? Moreover they are charging for access to the models.

Sure they are. Surf around on HuggingFace and you'll see dozens of open-weight models up to 120B parameters in size from for-profit US companies, freely downloadable (if not freely runnable, alas). Probably hundreds of them, at this point.

And a 3T parameter model is scheduled to be dropped by the Chinese on Monday.

They should certainly publish more, and if somebody were to argue that model weights trained by scraping copyrighted data should inherently be accessible to everyone, I'd be 100% in favor of that.

Re: What happened to TheNumbers.com

#119

Post by primitivesuave »

deepsun wrote: post #48
primitivesuave wrote: post #25

A couple years ago, I was running a website which allowed the public to view all the US government handouts to small businesses during the COVID-19 pandemic. It also tracked all the fraudulent loans being prosecuted by the DOJ, and allowed anyone to run structured queries over the public dataset. There was a "donate" button which took in ~$2k in donations over the lifetime of the site, and you could download the enti…

BigQuery has "public datasets", so users can even run complex SQL on it, but it's them who pays for it, not you. You only pay for data storage.

Thanks for that tip, this is exactly how I would do this if I had to do it from scratch. Just in case it's useful for anyone else: https://docs.cloud.google.com/bigquery/public-data

Re: What happened to TheNumbers.com

#120

Post by primitivesuave »

dpoloncsak wrote: post #64
deepsun wrote: post #48

Earlier quoted context omitted.

BigQuery has "public datasets", so users can even run complex SQL on it, but it's them who pays for it, not you. You only pay for data storage.

The crawlers would have still just hammered their site though, right?

Yes, if I wanted to put a nice HTML interface over the query results then I would still end up with the same problem where some combinatoric explosion of query parameters to the `/search` endpoint, most of which are cache misses, leads to many many TBs of network egress.
Post reply on HN