Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An update on Wayback Machine access

blog.archive.org

Post reply on HN
Print view View original

261–270 of 374 posts

Re: An update on Wayback Machine access

#261

Post by katatue »

xp84 wrote: post #22
Onavo wrote: post #2

Why not just offer a paid endpoint for the crawlers? It's not like the demand is going to go away anytime soon. It serves nobody except CloudFlare and hardware companies when one side set up blockers and the other side spend money putting VPN SDKs in consumer TVs. I am also curious how the (Russian?) paywall bypass mirror archive.is is doing given that they are probably subject to similar amounts of traffic.

My guess? Because even with a paid endpoint, the type of unscrupulous yahoo that is DDOSing IA today would probably still abuse the free endpoints because they can. The revenue that might come from a paid endpoint could help to scale up, but with how slow IA usually seems, I suspect there is an upper limit to how much traffic they can serve without a LOT more revenue. This is a major "this is why we can't have nice t…

IA might be large enough to earn consideration, but generally scrapers just don't care about being good citizens. I work in the GLAM space and we offer OAI-PMH interfaces for the harvesting of our collections data - which doesn't stop companies from preferring to scrape our website for worse (less complete, less structured, less standardized) data instead.

Re: An update on Wayback Machine access

#262

Post by simonjgreen »

simonw wrote: post #3

> Here’s what’s going on. The Internet Archive’s Wayback Machine has been hit by waves of high-volume automated traffic, and we’ve put protections in place to keep the service running. I'm pretty certain this is scrapers that are trying to workaround blocks on accessing original sites by hitting the Wayback Machine copy instead. Appalling behavior. In addition to the load it puts on this vital non-profit piece of Int…

Nearly every time a link is posted to HN to a site behind some form of wall, a high voted comment on the post will be a link to an archive site bypassing the owners wall. Bot owners are not the only ones routinely circumventing the choices of content owners.

Re: An update on Wayback Machine access

#263

Post by fulafel »

HDBaseT wrote: post #191
xacky wrote: post #118

The anti virus industry needs to crack down on crawler and proxy malware, plus ISPs FINALLY need to replace CGNATs with iov6 to stop crawlers banning everyone behind a NAT.

How exactly does IPv6 "stop crawlers". If anything, it will make it harder to block due to the vastness of the IPv6 address space.

CGNAT makes all ISP users appear to come from one v4 address, so blocking by v4 address becomes unworkable.

Re: An update on Wayback Machine access

#264

Post by x______________ »

schnebbau wrote: post #255
LoganDark wrote: post #252

Earlier quoted context omitted.

archive.today uses clients to perform DDoS, I would not recommend using their site.

[flagged]

Sure it is! This has been going on for years and global attention was gained at the beginning of this one.[0]

Wikipedia deprecates Archive.today, starts removing archive links (arstechnica.com) 616 points by nobody9999 6 months ago | hide | past | favorite | 368 comments

0 https://news.ycombinator.com/item?id=47092006

Re: An update on Wayback Machine access

#265

Post by dhx »

bradly wrote: post #77
aaron_m04 wrote: post #73

Earlier quoted context omitted.

robots.txt?

Has it been settled whether robots.txt applies to user-driven chat sessions and if things like the crawl delay should be applied to say an end-user, an ip address, a harness provider, etc? My understanding is robots.txt is more for training exclusions, but less so for agent work.

robots.txt was only intended to help search index crawlers not get stuck in endless crawl loops for badly designed websites.

What you suggest is explicitly not a purpose of robots.txt per RFC9309[1]:

"These rules are not a form of access authorization."

HTTP 429 and HTTP 403 are what servers are meant to return to clients to slow them down or tell them to stop doing something without having first gained authorisation.

[1] https://datatracker.ietf.org/doc/html/rfc9309#section-1

Re: An update on Wayback Machine access

#266

Post by progval »

bonestamp2 wrote: post #91
subarctic wrote: post #61

Earlier quoted context omitted.

What if they charged money? Is it something you'd pay for?

I was thinking the same thing... paid access for high volume users or scrapers could actually help fund the non-profit. Maybe let website owners decide which scrapers are allowed to use their content, or allow them to get paid for use of it. If news and other sites were getting paid, maybe they could go back to optimizing for good content instead of clicks.

They already provide this service at https://archive-it.org/archive-it/ though for some reason they don't seem to publicize it. Some info at https://help.archive.org/help/archive-it-information/ as well.

Re: An update on Wayback Machine access

#268

Post by dewey »

grumbelbart2 wrote: post #258
usr1106 wrote: post #242

Earlier quoted context omitted.

Sorry, not following. I thought the user agent is a string that the caller can set to anything. There is no immediate, reliable way to tell whether the string is correct. Yeah, a real browser would produce certain patterns and never certain others. So in some cases one could clearly say it's not a human using a browser. But a scraper could also make efforts to mimic human browsing. Mostly the frequency of requests ca…

Google's scraper bot at least used to be behind IPs that you could identify via reverse-then-forward DNS. Not sure if that is still up to date, though. https://developers.google.com/search/blog/2006/09/how-to-ver...

Yep, verifying the IPs is still the way to go. You often see websites that do it wrong when you set your user agent to Google Bot and they give you a different version of the page without validating that.

Re: An update on Wayback Machine access

#269

Post by nubinetwork »

danbolt wrote: post #166
stickfigure wrote: post #44

Earlier quoted context omitted.

Plenty of threads on HN about this, Anubis does not work.

I’ve read a few different experiences with hosts having had success with Anubis to cut down on excessive scraping. One that comes to mind is the Dolphin project.[1] I’ve read a few of those threads, but often it’s people at cross-purposes with the goals of Anubis. Is there a chance you could clarify the “not working” bit? [1] https://dolphin-emu.org/blog/2025/06/04/dolphin-progress-rep...

> Is there a chance you could clarify the “not working” bit?

I see anubis, 90% of the time I close the tab before it finishes.

Re: An update on Wayback Machine access

#270

Post by landgenoot »

imglorp wrote: post #130
Analemma_ wrote: post #81

Earlier quoted context omitted.

Micropayments would solve all the problems except for the problem that people absolutely loathe micropayments. Like, vein-popping furiously hate them. Whenever the topic of micropayments for internet content comes up, a bunch of people start talking about payment processors and their floor on prices, and so on. That's not wrong, but it can be designed around and I think it's a scapegoat to avoid confronting the fact…

It doesn't need to be crypto, or payment processor based. My ideal experience would be I load $20 into the browser somewhere like a wallet in one block (that could be a payment processor step). If I visit a participating page, it decrements my wallet $.01 or whatever. The downside is the possibility of abuse and tracking by governments, which would have to be handled at the source, not the symptom.

I worked on this before. You can solve the privacy issue using "statistical payments", by lack of a better word.

You visit website A,A,A,B,C,D,A,A

At the end of the month, you send your entire 20$ randomly to one of the websites you visited.

This will level out everyone's contribution and reward websites with lots of traffic. It eliminates the need for micropayments.

Post reply on HN