Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Internet Archive Infrastructure

archive.org

Post reply on HN
Print view View original

31–40 of 114 posts

Re: Internet Archive Infrastructure

#31

Post by ignoramous »

tkgally wrote: post #5

This is a recent video presentation by Jonah Edwards, who runs the Core Infrastructure Team at the Internet Archive. He explains the IA’s server, storage, and networking infrastructure, and then takes questions from other people at the Archive. I found it all interesting. But the main takeaway for me is his response to Brewster Kahle’s question, beginning at 13:06, about why the IA does everything in-house rather tha…

Speaking of Infrastructure, it is amazing that the initial set of the Apache BigData projects started at Internet Archive [0] whilst Alexa Internet, a startup Brewster Kahle sold to Amazon in 1999, formed the basis of Alexa Web Information Service, one of the first ''AWS'' products [1] which is still up? https://aws.amazon.com/awis/

[0] http://radar.oreilly.com/2015/04/coming-full-circle-with-big...

[1] https://web.stanford.edu/class/ee204/Publications/Amazon-EE3...

Re: Internet Archive Infrastructure

#33

Post by Nextgrid »

janjim wrote: post #28
tkgally wrote: post #5

This is a recent video presentation by Jonah Edwards, who runs the Core Infrastructure Team at the Internet Archive. He explains the IA’s server, storage, and networking infrastructure, and then takes questions from other people at the Archive. I found it all interesting. But the main takeaway for me is his response to Brewster Kahle’s question, beginning at 13:06, about why the IA does everything in-house rather tha…

Curious about the cost. Does that already include manpower and various acquisition cost of constructing their internal network (hardware, fiber link between site)? I guess the biggest downside is the speed of scaling that they can do. As it is limited by how fast they can purchase and install new storage device. But with the use case of Internet Archive, that shouldn't matter much.

I've seen this argument a lot but I'm not sure how well it holds. The price for performance ratio on cloud providers is so poor that you can overprovision in advance (to mitigate the extra delay involved in adding extra hardware) and still come out ahead.

Also, bare-metal doesn't necessarily mean owning the hardware. You can rent it too. There are providers that provide bare-metal in one-click and sometimes available within minutes.

Re: Internet Archive Infrastructure

#34

Post by ignoramous »

fireattack wrote: post #16

Probably a good place to ask: is there a reason why Wayback Machine's archives often take forever to load? Most of content on IA loads pretty fast, so WB is a notable exception.

Wayback Machine started life at Amazon's Search project. May be it never recovered from that :) https://archive.is/2B2ts

Re: Internet Archive Infrastructure

#35

Post by beckman466 »

kilroy123 wrote: post #27
fireattack wrote: post #16

Probably a good place to ask: is there a reason why Wayback Machine's archives often take forever to load? Most of content on IA loads pretty fast, so WB is a notable exception.

This is my big gripe as well, it's so painfully slow. Still, I strongly support the work they do and think it's very important work. I also think they do a good job for their size and resources.

I'm surprised to read this and OP's comment, for me it's always quite fast. Are there specific websites your request - are they media heavy?

Re: Internet Archive Infrastructure

#36

Post by chemicalnovae »

philjohn wrote: post #26
vermilingua wrote: post #20

Earlier quoted context omitted.

And disk life.

perversely, wouldn't spinning drives up and down impact drive life?

Probably yes, but only if it was happening a lot; I don’t know what the cross-over point would be though that you’re better off just keeping the drives spinning...

Re: Internet Archive Infrastructure

#37

Post by tkgally »

jasoncartwright wrote: post #32

I didn't hear it mentioned, but is there a backup outside the Bay Area?

According to an IA blog post in 2016 [1], they had partial backups in Egypt and the Netherlands then and were planning to establish a partial mirror in Canada as well. Perhaps someone at the IA can tell us what the current status of those mirrors is.

Brewster Kahle does mention in the video (around 26:50) that one reason they use paired storage is so that the two disks in a pair can be in different countries. He then goes on to praise “Linux and the wonder of open source”; he recently blogged about that, too [2].

[1] http://blog.archive.org/2016/12/03/faqs-about-the-internet-a...

[2] http://blog.archive.org/2021/02/04/thank-you-ubuntu-and-linu...

Re: Internet Archive Infrastructure

#38

Post by 88 »

lprd wrote: post #22

This is awesome! I love seeing companies run their own infrastructure. I wonder if they are using ZFS or just traditional RAID?

This is mentioned in the video, but they don’t use any form of RAID, just paired/mirrored drives in different physical locations.

This is preferred for its simplicity and performance, and if I were in their position I would do the same thing.

Re: Internet Archive Infrastructure

#39

Post by 88 »

bestboy wrote: post #18

If I understood it correctly, then they are using simple physical disk mirrors for redundancy. To me that seems like a huge waste of disk space. Parity based redundancy schemes like RAID-Z3 are way more space efficient. I do understand that parity based schemes need more time to heal/rebuild on drive replacements, but that does not seem to outweigh the huge amount of wasted disk space IMHO.

The paired disks are in a different physical location so they also provide a degree of geographic redundancy.

Short of splitting a RAID array across two physical locations (a terrible idea), your proposal would require them to run mirrored RAID arrays in both locations.

This would give them greater redundancy, but would be a less efficient use of raw disk space than their current solution. It would also be more complex and difficult to maintain, and have performance impacts.

Re: Internet Archive Infrastructure

#40

Post by tutfbhuf »

walrus01 wrote: post #13
benaadams wrote: post #10

Earlier quoted context omitted.

I assume the 200PB of storage and 60Gbps egress bandwidth 24/365 they do would be _extremely_ pricy on AWS...

On the scale of big hosting operations, 60Gbps outbound is not that much. If you're buying full tables IP transit from major carriers at IX points, I've seen 10GbE for $700-900/mo, and 100GbE circuits for under $7k/month. Of course you wouldn't want to have just one transit, but I'm fairly sure if somebody said to me 'here's $20,000 a month to buy transit', on the west coast, it's within the realm of possible. Ideall…

It seems that companies can be too big for the cloud and too small for the cloud (don't need k8s). I wonder what's the sweet spot.
Post reply on HN