I've been looking into Ceph a lot recently and was just wondering if they used it. Apparently not. Perhaps too abstract given their value of simplicity.
Internet Archive Infrastructure
11–20 of 114 posts
Re: Internet Archive Infrastructure
#12This is a recent video presentation by Jonah Edwards, who runs the Core Infrastructure Team at the Internet Archive. He explains the IA’s server, storage, and networking infrastructure, and then takes questions from other people at the Archive. I found it all interesting. But the main takeaway for me is his response to Brewster Kahle’s question, beginning at 13:06, about why the IA does everything in-house rather tha…
I assume the 200PB of storage and 60Gbps egress bandwidth 24/365 they do would be _extremely_ pricy on AWS...
I have no clue how people can afford what AWS charges for bandwidth. I did the math once for migrating a project to AWS and the bandwidth alone costed 10x my entire current infrastructure for that project, which is something I run for free.
Re: Internet Archive Infrastructure
#13This is a recent video presentation by Jonah Edwards, who runs the Core Infrastructure Team at the Internet Archive. He explains the IA’s server, storage, and networking infrastructure, and then takes questions from other people at the Archive. I found it all interesting. But the main takeaway for me is his response to Brewster Kahle’s question, beginning at 13:06, about why the IA does everything in-house rather tha…
I assume the 200PB of storage and 60Gbps egress bandwidth 24/365 they do would be _extremely_ pricy on AWS...
On the scale of big hosting operations, 60Gbps outbound is not that much. If you're buying full tables IP transit from major carriers at IX points, I've seen 10GbE for $700-900/mo, and 100GbE circuits for under $7k/month. Of course you wouldn't want to have just one transit, but I'm fairly sure if somebody said to me 'here's $20,000 a month to buy transit', on the west coast, it's within the realm of possible.
Ideally of course they should be able to meet a fairly wide number of downstream eyeball ISPs at the major IX points in the bay area and offload a lot of traffic with settlement-free peering.
60Gbps outbound from AWS, Azure or GCP would be astronomically incredibly expensive.
Re: Internet Archive Infrastructure
#14I've been looking into Ceph a lot recently and was just wondering if they used it. Apparently not. Perhaps too abstract given their value of simplicity.
Check this experiment they did https://github.com/internetarchive/sandcrawler/blob/master/p... . This was just a part of the infrastructure.
Point is ceph & friends have a lot of overhead. Example in ceph: by default, a file in S3 layer is split into 4MB chunks, and each of those chunks is replicated or erasure-coded. Using the same erasure coding as wasabi,b2-cloud, which is 16+4=20 (or 17+3=20), each of those 4MB chunks is split into 20 shards of ~200KB each. Each of those shards ends up having ~512B to 4KB of metadata.
So from 10KB to 80KB of metadata for single 4MB chunk.
Re: Internet Archive Infrastructure
#15I've been looking into Ceph a lot recently and was just wondering if they used it. Apparently not. Perhaps too abstract given their value of simplicity.
Check this experiment they did https://github.com/internetarchive/sandcrawler/blob/master/p... . This was just a part of the infrastructure. Point is ceph & friends have a lot of overhead. Example in ceph: by default, a file in S3 layer is split into 4MB chunks, and each of those chunks is replicated or erasure-coded. Using the same erasure coding as wasabi,b2-cloud, which is 16+4=20 (or 17+3=20), each of those 4MB c…
The same 4MB chunks in seaweedfs, would be ~40B * 20 = ~800Bytes.
Note: seaweedfs doesn't actually support 16+4, it's set in 10+4 in source-code. But the architecture makes the low overhead possible.
Re: Internet Archive Infrastructure
#16Probably a good place to ask: is there a reason why Wayback Machine's archives often take forever to load?
Most of content on IA loads pretty fast, so WB is a notable exception.
Re: Internet Archive Infrastructure
#17Probably a good place to ask: is there a reason why Wayback Machine's archives often take forever to load? Most of content on IA loads pretty fast, so WB is a notable exception.
I get the impression Wayback Machine data is stored in powered down drives and they only spin them up when someone accesses the data. That would explain the several second delay and it'd make sense that an archive wouldn't need 95% of its data ready to go at a moments notice since that'd be a terrible waste of power.
Re: Internet Archive Infrastructure
#18If I understood it correctly, then they are using simple physical disk mirrors for redundancy. To me that seems like a huge waste of disk space. Parity based redundancy schemes like RAID-Z3 are way more space efficient.
I do understand that parity based schemes need more time to heal/rebuild on drive replacements, but that does not seem to outweigh the huge amount of wasted disk space IMHO.
Re: Internet Archive Infrastructure
#19If I understood it correctly, then they are using simple physical disk mirrors for redundancy. To me that seems like a huge waste of disk space. Parity based redundancy schemes like RAID-Z3 are way more space efficient. I do understand that parity based schemes need more time to heal/rebuild on drive replacements, but that does not seem to outweigh the huge amount of wasted disk space IMHO.
There are tables for bandwidth and iops vs differing raid levels, mirroring is generally the best.
Re: Internet Archive Infrastructure
#20Probably a good place to ask: is there a reason why Wayback Machine's archives often take forever to load? Most of content on IA loads pretty fast, so WB is a notable exception.
I get the impression Wayback Machine data is stored in powered down drives and they only spin them up when someone accesses the data. That would explain the several second delay and it'd make sense that an archive wouldn't need 95% of its data ready to go at a moments notice since that'd be a terrible waste of power.
And disk life.