Might be worth adding [video] to the title.
Internet Archive Infrastructure
21–30 of 114 posts
Re: Internet Archive Infrastructure
#22This is awesome! I love seeing companies run their own infrastructure. I wonder if they are using ZFS or just traditional RAID?
Re: Internet Archive Infrastructure
#23If I understood it correctly, then they are using simple physical disk mirrors for redundancy. To me that seems like a huge waste of disk space. Parity based redundancy schemes like RAID-Z3 are way more space efficient. I do understand that parity based schemes need more time to heal/rebuild on drive replacements, but that does not seem to outweigh the huge amount of wasted disk space IMHO.
There are tables for bandwidth and iops vs differing raid levels, mirroring is generally the best.
True, but I do not consider nearline archiving to be a workload that requires lots of IOPS.
Re: Internet Archive Infrastructure
#24Earlier quoted context omitted.
I assume the 200PB of storage and 60Gbps egress bandwidth 24/365 they do would be _extremely_ pricy on AWS...
I have no clue how people can afford what AWS charges for bandwidth. I did the math once for migrating a project to AWS and the bandwidth alone costed 10x my entire current infrastructure for that project, which is something I run for free.
For lots of orgs I’ve seen it creeps up slowly until you’re paying 10-50x of full transit without any peers but by that point you’re too locked in to do anything
Re: Internet Archive Infrastructure
#25Earlier quoted context omitted.
I assume the 200PB of storage and 60Gbps egress bandwidth 24/365 they do would be _extremely_ pricy on AWS...
I have no clue how people can afford what AWS charges for bandwidth. I did the math once for migrating a project to AWS and the bandwidth alone costed 10x my entire current infrastructure for that project, which is something I run for free.
Depends on whether bandwidth is important to what you're doing. In many applications it isn't so even at the inflated prices charged by AWS et al don't really matter in the context of other expenses.
Re: Internet Archive Infrastructure
#26Earlier quoted context omitted.
I get the impression Wayback Machine data is stored in powered down drives and they only spin them up when someone accesses the data. That would explain the several second delay and it'd make sense that an archive wouldn't need 95% of its data ready to go at a moments notice since that'd be a terrible waste of power.
And disk life.
perversely, wouldn't spinning drives up and down impact drive life?
Re: Internet Archive Infrastructure
#27Probably a good place to ask: is there a reason why Wayback Machine's archives often take forever to load? Most of content on IA loads pretty fast, so WB is a notable exception.
This is my big gripe as well, it's so painfully slow.
Still, I strongly support the work they do and think it's very important work. I also think they do a good job for their size and resources.
Re: Internet Archive Infrastructure
#28This is a recent video presentation by Jonah Edwards, who runs the Core Infrastructure Team at the Internet Archive. He explains the IA’s server, storage, and networking infrastructure, and then takes questions from other people at the Archive. I found it all interesting. But the main takeaway for me is his response to Brewster Kahle’s question, beginning at 13:06, about why the IA does everything in-house rather tha…
Curious about the cost. Does that already include manpower and various acquisition cost of constructing their internal network (hardware, fiber link between site)?
I guess the biggest downside is the speed of scaling that they can do. As it is limited by how fast they can purchase and install new storage device. But with the use case of Internet Archive, that shouldn't matter much.
Re: Internet Archive Infrastructure
#29[deleted]
Re: Internet Archive Infrastructure
#30[deleted]