Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Internet Archive Infrastructure

archive.org

Post reply on HN
Print view View original

51–60 of 114 posts

Re: Internet Archive Infrastructure

#51

Post by antongribok »

ddorian43 wrote: post #14
acidburnNSA wrote: post #11

I've been looking into Ceph a lot recently and was just wondering if they used it. Apparently not. Perhaps too abstract given their value of simplicity.

Check this experiment they did https://github.com/internetarchive/sandcrawler/blob/master/p... . This was just a part of the infrastructure. Point is ceph & friends have a lot of overhead. Example in ceph: by default, a file in S3 layer is split into 4MB chunks, and each of those chunks is replicated or erasure-coded. Using the same erasure coding as wasabi,b2-cloud, which is 16+4=20 (or 17+3=20), each of those 4MB c…

You will always have overhead.

In the video they mention storing everything in regular files on the filesystem. A regular filesystem would have inode overhead as well. XFS by default has 512 byte inodes (it can be more if you format it with bigger inodes, like you would for Ceph's Filestore backend).

For a lot of workloads Ceph's default erasure coding scheme (and Bluestore) would still be a lot more efficient than mirroring a file on top of a regular filesystem.

Re: Internet Archive Infrastructure

#52

Post by ahrs »

If anyone from the IA reads this, are there any plans for IPv6 support? Archiving IPv4 for future generations is an important goal so it's perhaps fitting that in 2021 the IA is still running a historical Internet Protocol but it would nice to have IPv6 support for those running IPv6-only networks.

Re: Internet Archive Infrastructure

#54

Post by notacoward »

bestboy wrote: post #18

If I understood it correctly, then they are using simple physical disk mirrors for redundancy. To me that seems like a huge waste of disk space. Parity based redundancy schemes like RAID-Z3 are way more space efficient. I do understand that parity based schemes need more time to heal/rebuild on drive replacements, but that does not seem to outweigh the huge amount of wasted disk space IMHO.

Besides the cross-DC issue others have mentioned, erasure coding everything can also exacerbate CPU or memory bottlenecks. Not sure if this is an issue for IA, but on my last project data would be initially replicated and then transparently converted to erasure codes after some time. I believe that some other exabyte-scale storage systems work similarly.

Re: Internet Archive Infrastructure

#55

Post by owenmarshall »

walrus01 wrote: post #13
benaadams wrote: post #10

Earlier quoted context omitted.

I assume the 200PB of storage and 60Gbps egress bandwidth 24/365 they do would be _extremely_ pricy on AWS...

On the scale of big hosting operations, 60Gbps outbound is not that much. If you're buying full tables IP transit from major carriers at IX points, I've seen 10GbE for $700-900/mo, and 100GbE circuits for under $7k/month. Of course you wouldn't want to have just one transit, but I'm fairly sure if somebody said to me 'here's $20,000 a month to buy transit', on the west coast, it's within the realm of possible. Ideall…

IME with cloud in the small and in the large: network prices are artificially high on the cloud providers and are very easy to get discounted if you are a big spender.

Re: Internet Archive Infrastructure

#56

Post by niea_11 »

jmiskovic wrote: post #49

I see incredible value in IA collection of books, videos and software. OTOH I'm puzzled by lack of organization. Take for example this newer document: https://archive.org/details/manualzilla-id-5695071 The document has horrible name and useless tags, and the content seems to be only section 2 of some SW manual. How would I ever hope to find it if I needed that exact document? Obviously such huge archive cannot be cat…

To be fair the linked document was uploaded just today to a collection that seems be considered a "waystation" collection so probably the document will be moved later to a permanent collection.

And I think they have bots that process the uploaded documents to do OCR and create previews.

Re: Internet Archive Infrastructure

#57

Post by johannes1234321 »

polote wrote: post #44
charcircuit wrote: post #12

Earlier quoted context omitted.

I have no clue how people can afford what AWS charges for bandwidth. I did the math once for migrating a project to AWS and the bandwidth alone costed 10x my entire current infrastructure for that project, which is something I run for free.

Because people have nothing to compare their AWS cost to. They don't know how much it would cost them to host their service outside of AWS. And it is not only a cost comparison. You need different kind of people to manage in-house vs cloud, not that you need less or more, just different skills

Not only that: It is really hard to predict AWS cost. So many variables go in. And starting with a small side project in AWS is easy, and then each additional step is a small step ...

Re: Internet Archive Infrastructure

#58

Post by coldpie »

beckman466 wrote: post #35
kilroy123 wrote: post #27

Earlier quoted context omitted.

This is my big gripe as well, it's so painfully slow. Still, I strongly support the work they do and think it's very important work. I also think they do a good job for their size and resources.

I'm surprised to read this and OP's comment, for me it's always quite fast. Are there specific websites your request - are they media heavy?

I picked a random article from my browser's history, linked below. Just loading the snapshot year pages took about 10 seconds, then the snapshot hover took another 10 seconds. Finally, fully loading the snapshot page took about 50 seconds. So that's roughly 90 seconds to go from inserting a URL into the search bar and actually having the rendered page. Not unacceptable, but certainly slow by modern standards.

http://web.archive.org/web/20120801000000*/http://blogs.msdn...

Re: Internet Archive Infrastructure

#59

Post by ajdude »

chimbosonic wrote: post #53

Does the IA have Data sites that are not in SF? When he shows the map of sites they all seem very close to each other and a natural disaster could wipe out alot of the archive.

They are mostly all in California, though not entirely in SF. They send some of their data to other parts of the world too, but I'm concerned that the don't have the redundancy needed. There was a project in attempt to back up the internet archive, but it became unmaintained in 2019 and only about 200tb were actually being backed up http://iabak.archiveteam.org/

Re: Internet Archive Infrastructure

#60

Post by Sebb767 »

benaadams wrote: post #10
tkgally wrote: post #5

This is a recent video presentation by Jonah Edwards, who runs the Core Infrastructure Team at the Internet Archive. He explains the IA’s server, storage, and networking infrastructure, and then takes questions from other people at the Archive. I found it all interesting. But the main takeaway for me is his response to Brewster Kahle’s question, beginning at 13:06, about why the IA does everything in-house rather tha…

I assume the 200PB of storage and 60Gbps egress bandwidth 24/365 they do would be _extremely_ pricy on AWS...

To be fair here, when you're pouring that much money into AWS you probably have a better contract and can negotiate the price down quite a bit. Additionally, you could use CloudFront to further reduce your bandwidth costs.

That's not to say that it wouldn't be incredibly expensive, but probably far less than what you see on the pricing page.

Post reply on HN