Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An update on Wayback Machine access

blog.archive.org

Post reply on HN
Print view View original

11–20 of 374 posts

Re: An update on Wayback Machine access

#13

Post by BeetleB »

Wow, but I wonder if there's more to it.

I've not been able to access web.archive.org from my work computer - I always get the 429 error.

But I then pull out my phone and can access it just fine. All along I was assuming my company was blocking it. Still weird that it happens every time from my work PC and never from my home one.

Re: An update on Wayback Machine access

#15

Post by faefox »

croes wrote: post #4
Onavo wrote: post #2

Why not just offer a paid endpoint for the crawlers? It's not like the demand is going to go away anytime soon. It serves nobody except CloudFlare and hardware companies when one side set up blockers and the other side spend money putting VPN SDKs in consumer TVs. I am also curious how the (Russian?) paywall bypass mirror archive.is is doing given that they are probably subject to similar amounts of traffic.

It’s one thing to archive other companies content, it’s another to sell the access to it

Yeah, who does the Internet Archive think it is, (insert literally any AI company here)?

Re: An update on Wayback Machine access

#16

Post by luckylion »

simonw wrote: post #3

> Here’s what’s going on. The Internet Archive’s Wayback Machine has been hit by waves of high-volume automated traffic, and we’ve put protections in place to keep the service running. I'm pretty certain this is scrapers that are trying to workaround blocks on accessing original sites by hitting the Wayback Machine copy instead. Appalling behavior. In addition to the load it puts on this vital non-profit piece of Int…

What sites would they be targeting? Generic "just give me anything"? Whenever I check regular sites on IA, the coverage is spotty -- they'll have the homepage and a few important pages, but it quickly fizzles out.

Very understandable, you can't store all 15000 pages of any random website and update them etc etc, but that makes them pretty useless for indirect scraping because you usually don't want a tiny taste, you want everything.

Re: An update on Wayback Machine access

#17

Post by timpera »

I really appreciate the Archive team's efforts to make the Wayback Machine more responsive, and have donated a few times to support them.

Unfortunately, the restrictions have been way too strict for the last few months: from my residential IP, simply moving the mouse on the calendar for a specific URL is enough to get stuck on 429 error messages for a while; and from corporate ISPs (for example, on airport WiFi), you often can't access the WM at all. I hope they'll find a way to relax those.

Re: An update on Wayback Machine access

#20

Post by dotmanish »

BeetleB wrote: post #13

Wow, but I wonder if there's more to it. I've not been able to access web.archive.org from my work computer - I always get the 429 error. But I then pull out my phone and can access it just fine. All along I was assuming my company was blocking it. Still weird that it happens every time from my work PC and never from my home one.

Could be due to some scrapers from either your work ISP block, or the larger block which lends IPs to multiple workplaces.
Post reply on HN