Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What happened to TheNumbers.com

stephenfollows.com

Post reply on HN
Print view View original

61–70 of 218 posts

Re: What happened to TheNumbers.com

#61

Post by jtuple »

CamperBob2 wrote: post #29
podgietaru wrote: post #27

"One Reddit theory even suggested it was a deliberate rug pull designed to cripple the free site to push people towards paid products." I know this isn't really the point of the article, but I've been thinking about this a lot. I wonder if we're going to see more resources go this way. I used to publish little doo-dads as opensource software. Not because it was something that was legitimately ground breaking or anyth…

This attitude makes zero sense to me. You benefit from the "training data" just like everyone else does. If you don't, that's a problem with you, not a problem with AI models. AI solves exactly the meta-problem you describe: "I had a problem and needed to write a one-off doo-dad utility program to solve it." Now you can do something with your time besides writing pointless one-off doo-dads. As for monetizing the trai…

> Now you can do something with your time besides writing pointless one-off doo-dads

Writing software one-offs to scratch an itch was historically one of my most enjoyable past-times. AI trivializing that has been a very real theft of joy in my life.

Solving the actual problem was never the point, it was just motivation to do geek-out and craft some code.

AI is rapidly diminishing many interesting hobbies (coding, art, music, writing).

Having more free time when there's nothing fun nor exciting to do with it isn't really a benefit.

Re: What happened to TheNumbers.com

#62

Post by tptacek »

jaredwiener wrote: post #17
tengada1 wrote: post #7

Earlier quoted context omitted.

Yeah this seems like one of the easiest access models to create an excellent security model for -- basically just static publishing. Sounds like they were in need of a rewrite anyway! It probably seems daunting but to be honest this feels like a weekend's work at this point with LLM assistance. Not to be glib!

But it also destroys the business model behind the site. Sure, you could technically redesign to handle the bot traffic, but if the bot traffic is just taking the data and reducing any need for humans to visit the site, why is he putting in the effort to maintain the site?

The business model of the site is apparently private data sales, not ad revenue.

Re: What happened to TheNumbers.com

#64

Post by dpoloncsak »

deepsun wrote: post #48
primitivesuave wrote: post #25

A couple years ago, I was running a website which allowed the public to view all the US government handouts to small businesses during the COVID-19 pandemic. It also tracked all the fraudulent loans being prosecuted by the DOJ, and allowed anyone to run structured queries over the public dataset. There was a "donate" button which took in ~$2k in donations over the lifetime of the site, and you could download the enti…

BigQuery has "public datasets", so users can even run complex SQL on it, but it's them who pays for it, not you. You only pay for data storage.

The crawlers would have still just hammered their site though, right?

Re: What happened to TheNumbers.com

#65

Post by antisthenes »

II2II wrote: post #45
podgietaru wrote: post #27

"One Reddit theory even suggested it was a deliberate rug pull designed to cripple the free site to push people towards paid products." I know this isn't really the point of the article, but I've been thinking about this a lot. I wonder if we're going to see more resources go this way. I used to publish little doo-dads as opensource software. Not because it was something that was legitimately ground breaking or anyth…

> But now I'm really reluctant to give more stuff to the free web. Because the fact that it gets scraped and added to a pile of training data to later be monetized really rubs me the wrong way. The thing is, people were scraping and monetizing other people's websites long before the current LLM fad. The difference is the magnitude of the problem. > And I can see why maintainers of sites like this, or other free but i…

> The thing is, people were scraping and monetizing other people's websites long before the current LLM fad. The difference is the magnitude of the problem.

People doing it with a couple of machines and residential proxies versus Anthropic doing it with 2 data centers worth of machines.

Scale matters.

Re: What happened to TheNumbers.com

#66

Post by antisthenes »

hyperhello wrote: post #6

If the scrapers are going to get it anyway, put the data up as a zip somewhere.

> Read the Docs, a non-profit that hosts documentation for open-source software, who watched a single crawler download 73 terabytes of zipped HTML in one month, costing it over $5,000 in bandwidth

From the article.

Not the same site, but an example of the same issue.

Re: What happened to TheNumbers.com

#67

Post by pavel_lishin »

dpoloncsak wrote: post #64
deepsun wrote: post #48

Earlier quoted context omitted.

BigQuery has "public datasets", so users can even run complex SQL on it, but it's them who pays for it, not you. You only pay for data storage.

The crawlers would have still just hammered their site though, right?

I think the idea is that they could store the data in BigQuery, and point users of the site there.

Re: What happened to TheNumbers.com

#68

Post by pverheggen »

BeeOnRope wrote: post #49
primitivesuave wrote: post #25

A couple years ago, I was running a website which allowed the public to view all the US government handouts to small businesses during the COVID-19 pandemic. It also tracked all the fraudulent loans being prosecuted by the DOJ, and allowed anyone to run structured queries over the public dataset. There was a "donate" button which took in ~$2k in donations over the lifetime of the site, and you could download the enti…

Do you have any view on why the AI scrapers resulted in a heavier load than existing crawlers from eg search engines? Where they more exhaustive or more frequent?

There's been an explosion of vibe-coded scrapers that behave poorly and ignore robots.txt. Presumably OP disallowed crawling of the search endpoint for the reason stated (that crawling it would result in endless permutations of search filters.)

Re: What happened to TheNumbers.com

#69

Post by PunchyHamster »

pavel_lishin wrote: post #67
dpoloncsak wrote: post #64

Earlier quoted context omitted.

The crawlers would have still just hammered their site though, right?

I think the idea is that they could store the data in BigQuery, and point users of the site there.

The crawlers would have still just hammered their site though, right?

Re: What happened to TheNumbers.com

#70

Post by PunchyHamster »

zackmorris wrote: post #42
ethagnawl wrote: post #4

At the risk of oversimplifying things from a distance, this site -- especially the free, public-facing part of it -- seems like it would be an ideal candidate for a rewrite using static site generator/framework. That, coupled with a bot-aware CDN should keep them online at a reasonable cost for many years to come. Otherwise, I'm very curious to know more about their old and new architecture and what sorts of mitigati…

Ya this problem was solved 20 years ago with Varnish cache and Coral cache/CDN. I think what's really going on is that bots expose how underpowered web servers has gotten in recent years. In the 2000s, even poorly-architected PHP sites tended to serve about 200 requests per second, with 1000+ being common for static sites. I remember when Node.js came out and claimed that it could serve more like 100,000 RPS due to i…

> Ya this problem was solved 20 years ago with Varnish cache and Coral cache/CDN.

It was... if you are paying datacenter rates for the bandwidth

If you're paying cloud provider per GB pricing, nah, even text will add up if you happen to be targeted by a bunch of bots

> What went wrong is that nobody solved stuff like Russian doll caching in a general way separate from the programming language and database. We should have had ways to make dependency graphs using Etag headers as keys with real cache invalidation of dependent data.

we did that with nested ESI includes in Varnish so every "box" of content on the page was cached separately + some piping for invalidation, so if a given piece in the database was changed it sent invalidation to all nodes. There was also some grace so if the thing you wanted got updated RIGHT NOW you might get stale version while the new one is updated in background, and don't pay the latency cost

Post reply on HN