Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Thank you for helping us increase our bandwidth

blog.archive.org

Post reply on HN
Print view View original

81–90 of 207 posts

Re: Thank you for helping us increase our bandwidth

#81

Post by x3blah »

Archive.org works surprisingly well as a general purpose web proxy. Just prefix the URL, e.g., http://example.com, with https://web.archive.org/save/, e.g., https://web.archive.org/save/http://example.com

The aesthetic intrusiveness of the archive.org header and footer are minimal since I use a text-only browser that has no Javascript engine.

Sometimes I get "This url is not available on the live web or can not be archived." However this happens for only a surprisingly small minority of websites.

Rarely I find that /save is unsuccessful in which case I can still find past copies using something like

   curl -o 1.txt "https://web.archive.org/cdx/search/cdx?url=http://www.example.net&fl=timestamp,original" ;
   sed -i '/^[12][0-9]* h/!d;/^[12][0-9]* h/{s/^/http:\/\/web.archive.org\/web\//;s/ /\//;s/\r//;}' 1.txt
The limitation with past copies versus /save is that archive.org will not usually crawl past page one on websites with many successive pages, e.g., http://example.com/?page=2, http://example.com/?page=3, etc.

Has anyone ever considered mirroring archive.org, or parts of it, to other geographic locations.

Could this be done. Why or why not.

Re: Thank you for helping us increase our bandwidth

#82

Post by manquer »

tgsovlerkhgsel wrote: post #34

They increased the bandwidth and traffic immediately filled the new capacity. While this could be just a faster web site attracting more users / users were giving up due to slow loads before, it could also indicate that they have a lot of automated traffic that will consume whatever resources are available and can be throttled without endangering their mission. I hope they'll look into it, find a way to identify that…

It could also be they are limited by the capacity, For me Wayback machine is always slow , although my bandwidth is otherwise not been a bottleneck. Perhaps increased capacity is getting utilized by both increased usage and higher performance of their services .

Re: Thank you for helping us increase our bandwidth

#83

Post by fomine3 »

EE84M3i wrote: post #23
SteveNuts wrote: post #3

I'm really surprised they don't use more CDN for this. Anyone know the reason why it isn't served by something like cloudflare?

The geographic distribution benefits for archive.org seem minimal to me - for the actual content high latency isn't a big deal.

It's really benefiticial for outside of US.

Re: Thank you for helping us increase our bandwidth

#84

Post by [deleted] »

bzb3 wrote: post #13
chx wrote: post #7

I have revoked my donations until they close their pirate library. Sorry but let's call a spade a spade. They admit: > multiple readers can access a digital book simultaneously with the only caveat being it is borrowed for two weeks but let's face it, most value of a book comes from its first reading -- and what stops you from "borrowing" it again, anyways. It's been a gigantic disappointment for me to see them do th…

Culture wants to be free.

[deleted]

Re: Thank you for helping us increase our bandwidth

#85

Post by Endlessly »

microcolonel wrote: post #68
britmob wrote: post #64

Earlier quoted context omitted.

There is http://iabak.archiveteam.org , but it’s not exactly large.

If I'm reading that correctly, it would only cost a bit over 500 bucks a month to host that whole archive on BackBlaze B2. Furthermore it would not be so hard to translate Archive.org items to IPFS objects, if there were an effort to pin a significant number of them to storage and network.

Since numbers are not 100% clear...

(50 petabytes * 0.2% = 100 terabytes)

[$0.005 ($/GB/Month) BackBlaze cost]

[(50 petabytes) / (1 gigabyte) = 50,000,000]

(50,000,000 * $0.005 = $250,000 US$)

—————

Meaning based on my numbers, that is $250,000 USD a month to host 50 petabytes of data on BackBlaze.

Re: Thank you for helping us increase our bandwidth

#86

Post by dylz »

SteveNuts wrote: post #15
adtac wrote: post #5

Earlier quoted context omitted.

Please don't make the whole internet basically cloudflare. they've banned my VPN endpoint (Hetzner server) and as a result a huge chunk of websites already don't work for me despite me having done nothing wrong. I've heard reports of Tor users being restricted as well.

Okay, any one of the other CDNs then. It seems like the bandwidth costs and equipment are prohibitively expensive for them. There must be a reason they don't use them which is what I want to know.

Any of the other CDNs are absurdly prohibitively expensive, pay-per-GB, etc.

And would still have to download from their source.

Re: Thank you for helping us increase our bandwidth

#87

Post by underwater »

endgame wrote: post #21
makomk wrote: post #16

Earlier quoted context omitted.

Governments aren't the only ones overwriting old information with new either. The BBC has developed an annoying habit of completely overwriting old articles about Covid-19 with new, semi-related ones, meaning that the only way to see what they were saying earlier in the pandemic is through sites like the Internet Archive. There's also at least one outright correction that only seems to exist on the Internet Archive n…

Here's another one: "some of our old tweets were wrong so we just quietly deleted them lol": https://twitter.com/voxdotcom/status/1242537366620966912

This is necessary on Twitter. It's easy for fake or incorrect news to spread, you can't edit it and updates/self-replies are often hidden.

Re: Thank you for helping us increase our bandwidth

#88

Post by EdJiang »

dmd wrote: post #71
britmob wrote: post #64

Earlier quoted context omitted.

There is http://iabak.archiveteam.org , but it’s not exactly large.

The IA is about 50 PB. IABAK stores 100 TB, or 0.2% of it.

You can get an 8TB HDD for $150 right now. That's 6250 drives. That's about $1MM in drives, which doesn't sound that cost-prohibitive. Obviously that's not the whole cost since you need to pay for bandwidth, replication and other infrastructure like the host node, but it sounds like something that could be even hosted by a number of volunteers on r/datahoarder or r/homelab.

I also remember reading about Sia on HN, which is a dapp that pays hosts to store data and distributes it. Looking at the going rates on Sia ($1.45/TB/mo), that's $870k/yr. That's ~10% of the IA budget (which is only $10MM/yr, which sounds very efficient!) but shows that the order of magnitude is not that crazy.

Re: Thank you for helping us increase our bandwidth

#90

Post by ramraj07 »

Endlessly wrote: post #85
microcolonel wrote: post #68

Earlier quoted context omitted.

If I'm reading that correctly, it would only cost a bit over 500 bucks a month to host that whole archive on BackBlaze B2. Furthermore it would not be so hard to translate Archive.org items to IPFS objects, if there were an effort to pin a significant number of them to storage and network.

Since numbers are not 100% clear... (50 petabytes * 0.2% = 100 terabytes) [$0.005 ($/GB/Month) BackBlaze cost] [(50 petabytes) / (1 gigabyte) = 50,000,000] (50,000,000 * $0.005 = $250,000 US$) ————— Meaning based on my numbers, that is $250,000 USD a month to host 50 petabytes of data on BackBlaze.

That's a fraction of the AWS bills for many startups arguably doing absolutely nothing
Post reply on HN