What happened to TheNumbers.com
stephenfollows.com
What happened to TheNumbers.com
1–10 of 218 posts
Re: What happened to TheNumbers.com
#2Sorry! Just yesterday many of us decided that AI scraping isn't a real problem, and anytime it is blamed, it's a cover for something else.
Re: What happened to TheNumbers.com
#3Ironic that polymarkets were being advertised as helping society make better predictions.
Re: What happened to TheNumbers.com
#4At the risk of oversimplifying things from a distance, this site -- especially the free, public-facing part of it -- seems like it would be an ideal candidate for a rewrite using static site generator/framework. That, coupled with a bot-aware CDN should keep them online at a reasonable cost for many years to come.
Otherwise, I'm very curious to know more about their old and new architecture and what sorts of mitigation/scaling strategies they've started using to keep the site online.
Re: What happened to TheNumbers.com
#5Its just too easy for a technically minded bored person to produce slop that hammers websites. There needs to be a penalty for this.
Re: What happened to TheNumbers.com
#6If the scrapers are going to get it anyway, put the data up as a zip somewhere.
Re: What happened to TheNumbers.com
#7At the risk of oversimplifying things from a distance, this site -- especially the free, public-facing part of it -- seems like it would be an ideal candidate for a rewrite using static site generator/framework. That, coupled with a bot-aware CDN should keep them online at a reasonable cost for many years to come. Otherwise, I'm very curious to know more about their old and new architecture and what sorts of mitigati…
Yeah this seems like one of the easiest access models to create an excellent security model for -- basically just static publishing. Sounds like they were in need of a rewrite anyway!
It probably seems daunting but to be honest this feels like a weekend's work at this point with LLM assistance. Not to be glib!
Re: What happened to TheNumbers.com
#8I think people are missing one of the points of the article. It's not just that agents are hammering the site, it's that there might be lurking vulnerabilities that allow malicious usage, which is why it went down, then came back up with a fraction of the data and a reduced design.
The article says (speculates?) that malicious users are trying to get privileged access for an edge in prediction market betting. From the article:
> If you could see The Numbers data before everyone else, every single week, you would have a significant edge over all the other traders - learning the answers slightly ahead of publication would allow you to front-run the trades.
Re: What happened to TheNumbers.com
#9I always liked this site, but reading this and seeing the anger about expecting the site maintainer to do things for you is repulsive. Frankly, if he wanted to pull his site down with no notice that is perfectly within his right. It was/is his site. He doesn't owe anyone a .tar.gz either. His work.
Re: What happened to TheNumbers.com
#10[deleted]