Why not put it in S3 with downloader pays?
As a public resource the hope is for Wayback to be free to access. I imagine putting up a paywall would be their last resort.
An update on Wayback Machine access
41–50 of 374 posts
Re: An update on Wayback Machine access
#42[flagged]
Re: An update on Wayback Machine access
#43Re: An update on Wayback Machine access
#44I wonder if they'll end up behind Anubis at some point. I'm surprised it hasn't happened already.
Plenty of threads on HN about this, Anubis does not work.
Re: An update on Wayback Machine access
#45AI companies should pay billions to wayback machine for access
I think that'd raise serious copyright concerns, if the Wayback machine started selling other people's intellectual property.
Re: An update on Wayback Machine access
#46Why not just offer a paid endpoint for the crawlers? It's not like the demand is going to go away anytime soon. It serves nobody except CloudFlare and hardware companies when one side set up blockers and the other side spend money putting VPN SDKs in consumer TVs. I am also curious how the (Russian?) paywall bypass mirror archive.is is doing given that they are probably subject to similar amounts of traffic.
Micropayments would solve so many Internet problems. It's not too late to adopt.
Content creators could charge by page instead of depending on malware/ad/surveillance revenue. Spam is cut if there's a charge per mail. Scraping abuse goes away, along with a bunch of DDOS garbage.
The impact is a few cents per page or mail, negligible for a human. But if you're consuming a trillion pages per day, you'd reconsider.
Re: An update on Wayback Machine access
#47Re: An update on Wayback Machine access
#48Why not just offer a paid endpoint for the crawlers? It's not like the demand is going to go away anytime soon. It serves nobody except CloudFlare and hardware companies when one side set up blockers and the other side spend money putting VPN SDKs in consumer TVs. I am also curious how the (Russian?) paywall bypass mirror archive.is is doing given that they are probably subject to similar amounts of traffic.
> Why not just offer a paid endpoint for the crawlers?
Because then you're definitely violating US copyright law. There are four prongs of fair use analysis, and one of them is the "nature of the use." In this case, you'd be turning into a commercial use.
Re: An update on Wayback Machine access
#49> Here’s what’s going on. The Internet Archive’s Wayback Machine has been hit by waves of high-volume automated traffic, and we’ve put protections in place to keep the service running. I'm pretty certain this is scrapers that are trying to workaround blocks on accessing original sites by hitting the Wayback Machine copy instead. Appalling behavior. In addition to the load it puts on this vital non-profit piece of Int…
Just yesterday from my one of my sessions with Sol:
> Hacker News and the Rails forum are blocking the text fetcher, so I'm using the browser workflow to inspect the pages directly
Re: An update on Wayback Machine access
#50There was a feature on Amazon Web Services for a while, and I wish it was still there...
Downloader pays.
I make some content and upload it. When you want to download it, you pay Amazon the egress fees. And maybe I get to charge just a bit more, to help me with the Ingress, storage, content creation, etc.
I mean, I know that there's going to be problems with rate limiting, etc. And yes, we have those problems with LLM tokens today. But this just feels like such a useful thing that it baffles me that it doesn't exist already.