Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Internet Archive Infrastructure

archive.org

Post reply on HN
Print view View original

101–110 of 114 posts

Re: Internet Archive Infrastructure

#101

Post by ExtraE »

jolmg wrote: post #85
ExtraE wrote: post #77

Earlier quoted context omitted.

How does this work? I was under the impression that when I clicked a link in a webpage it just opened that link as if I'd typed it into my URL bar. How does jwz know were I'm clicking from?

https://en.wikipedia.org/wiki/HTTP_referer > The HTTP referer (a misspelling of referrer) is an optional HTTP header field that identifies the address of the webpage (i.e., the URI or IRI), which is linked to the resource being requested. By checking the referrer, the new webpage can see where the request originated. https://tools.ietf.org/html/rfc7231#page-45 > The Referer header field allows servers to generate bac…

Thanks!

Re: Internet Archive Infrastructure

#102

Post by lprd »

jonah-archive wrote: post #98
88 wrote: post #38

Earlier quoted context omitted.

This is mentioned in the video, but they don’t use any form of RAID, just paired/mirrored drives in different physical locations. This is preferred for its simplicity and performance, and if I were in their position I would do the same thing.

Exactly correct. I suspect that in the not-too-distant future we will need to move to multi-disk filesystem-level clustering on the storage nodes for some of the reasons laid out in the talk but it's not at all unlikely that we retain the "disconnected mirror" abstraction for redundancy. Additionally, as I alluded to in the video, we regularly end up going down into the details and when this happens being able to clo…

Hey Jonah,

Perhaps you already covered this in the video and I missed it, but I was wondering how the team went about hardware upgrades/disposal? I maintain a just few servers in my own homelab so upgrading is pretty trivial for me, but I can't imagine what its like on that scale. Also, which hypervisor are you using to manage all of those VMs?

Thanks again for the talk, it was very insightful and fun to watch!

Re: Internet Archive Infrastructure

#103

Post by GuB-42 »

samstave wrote: post #82
Ansil849 wrote: post #73

Earlier quoted context omitted.

> and greater confidence that their users are not being tracked. And that right there is why I continue to donate to IA. I am sick and tired of services offloading my data to destructive companies like Amazon.

Tracking should be STRICTLY illegal and ONLY acceptable with a verifiable OPT-IN and a transparency on to WHOM the data was sent/read-by/received. With STILL an option to selectively opt-out.

> Tracking should be STRICTLY illegal

And yet, the opposite is happening. Government agencies are happy to tap into user data. And sometimes it is mandatory to keep data depending on what you are doing and your country legislation.

AFAIK the US is among the countries where you have the least such requirements, but there are still some sectors where logging is mandatory, like financial services. Many other first world countries (never mind dictatorships) require ISPs to keep data for a year or more.

Re: Internet Archive Infrastructure

#104

Post by ddorian43 »

antongribok wrote: post #51
ddorian43 wrote: post #14

Earlier quoted context omitted.

Check this experiment they did https://github.com/internetarchive/sandcrawler/blob/master/p... . This was just a part of the infrastructure. Point is ceph & friends have a lot of overhead. Example in ceph: by default, a file in S3 layer is split into 4MB chunks, and each of those chunks is replicated or erasure-coded. Using the same erasure coding as wasabi,b2-cloud, which is 16+4=20 (or 17+3=20), each of those 4MB c…

You will always have overhead. In the video they mention storing everything in regular files on the filesystem. A regular filesystem would have inode overhead as well. XFS by default has 512 byte inodes (it can be more if you format it with bigger inodes, like you would for Ceph's Filestore backend). For a lot of workloads Ceph's default erasure coding scheme (and Bluestore) would still be a lot more efficient than m…

> For a lot of workloads Ceph's default erasure coding scheme (and Bluestore) would still be a lot more efficient than mirroring a file on top of a regular filesystem.

Yes that's correct, it's why Bluestore was created in the first place.

Re: Internet Archive Infrastructure

#105

Post by coldpie »

carapace wrote: post #87
coldpie wrote: post #58

Earlier quoted context omitted.

I picked a random article from my browser's history, linked below. Just loading the snapshot year pages took about 10 seconds, then the snapshot hover took another 10 seconds. Finally, fully loading the snapshot page took about 50 seconds. So that's roughly 90 seconds to go from inserting a URL into the search bar and actually having the rendered page. Not unacceptable, but certainly slow by modern standards. http://…

90s is better than ∞ (infinity) which is what it would be if the Archive didn't exist, eh?

The original comment was "the Archive is painfully slow", the reply to that was "for me it's always quite fast", so I gave it some actual hard data explaining that 90s is in fact quite slow and maybe even painfully so in 2021. I have absolutely no idea how you got from that discussion to a hypothetical where the Archive doesn't exist. Obviously 90s is better than it not existing. I even said in my comment that it was not unacceptable.

Why did you even make this comment?

Re: Internet Archive Infrastructure

#106

Post by carapace »

coldpie wrote: post #105
carapace wrote: post #87

Earlier quoted context omitted.

90s is better than ∞ (infinity) which is what it would be if the Archive didn't exist, eh?

The original comment was "the Archive is painfully slow", the reply to that was "for me it's always quite fast", so I gave it some actual hard data explaining that 90s is in fact quite slow and maybe even painfully so in 2021. I have absolutely no idea how you got from that discussion to a hypothetical where the Archive doesn't exist. Obviously 90s is better than it not existing. I even said in my comment that it was…

> Why did you even make this comment?

I find it crass to gripe about Archive being slow. It's still much better than nothing. If folks want it to be faster than can donate rather than gripe, eh?

Re: Internet Archive Infrastructure

#107

Post by jonah-archive »

lprd wrote: post #102
jonah-archive wrote: post #98

Earlier quoted context omitted.

Exactly correct. I suspect that in the not-too-distant future we will need to move to multi-disk filesystem-level clustering on the storage nodes for some of the reasons laid out in the talk but it's not at all unlikely that we retain the "disconnected mirror" abstraction for redundancy. Additionally, as I alluded to in the video, we regularly end up going down into the details and when this happens being able to clo…

Hey Jonah, Perhaps you already covered this in the video and I missed it, but I was wondering how the team went about hardware upgrades/disposal? I maintain a just few servers in my own homelab so upgrading is pretty trivial for me, but I can't imagine what its like on that scale. Also, which hypervisor are you using to manage all of those VMs? Thanks again for the talk, it was very insightful and fun to watch!

Thank you, so glad you enjoyed it!

We try to keep a regular upgrade cycle (to hold to our tight budget), typically with a tick-tock of adding new hardware (expanding within our existing footprint) and cycling out old hardware and drives as they reach the far end of the works/doesn't work spectrum. We have a local partner who takes care of some disposal for us, but we also have no shortage of physical storage space, so we will also accumulate (sometime intentionally -- nearly our entire "red box" deployment -- see "previous version": https://archive.org/web/petabox.php -- is packed into a shipping container. We don't like to throw things away!).

For a hypervisor, we use Ganeti (running over KVM). Because our fleet is so heterogenous we need to be able to control a lot of VM parameters in order to efficiently pack our computational resources, and Ganeti is kind of in a sweet spot for us in terms of providing a lot more tooling than a bunch of virsh scripts, and being much smaller than systems like OpenStack geared towards large, homogenous deployments).

Re: Internet Archive Infrastructure

#108

Post by jgowdy »

Honestly, I've lost a bit of confidence in the Internet Archive project's judgement and long term stability, given the risk they took in distributing books during Covid-19. I feel that they risked the entire project with massive (almost certainly fatal) copyright violation fines in order to distribute extra copies of books. That lawsuit is still pending and we don't know what the outcome of that lawsuit will be.

If they're willing to risk everything they've accomplished to date in order to issue a few extra copies of books, I don't see them surviving long term, nor do I feel comfortable donating to the project.

https://www.npr.org/2020/06/03/868861704/publishers-sue-inte...

Re: Internet Archive Infrastructure

#109

Post by beckman466 »

carapace wrote: post #106
coldpie wrote: post #105

Earlier quoted context omitted.

The original comment was "the Archive is painfully slow", the reply to that was "for me it's always quite fast", so I gave it some actual hard data explaining that 90s is in fact quite slow and maybe even painfully so in 2021. I have absolutely no idea how you got from that discussion to a hypothetical where the Archive doesn't exist. Obviously 90s is better than it not existing. I even said in my comment that it was…

> Why did you even make this comment? I find it crass to gripe about Archive being slow. It's still much better than nothing. If folks want it to be faster than can donate rather than gripe, eh?

Completely agree. It’s not like there’s alternatives to Archive.org available.

Re: Internet Archive Infrastructure

#110

Post by nikisweeting »

nodesocket wrote: post #63
tkgally wrote: post #5

This is a recent video presentation by Jonah Edwards, who runs the Core Infrastructure Team at the Internet Archive. He explains the IA’s server, storage, and networking infrastructure, and then takes questions from other people at the Archive. I found it all interesting. But the main takeaway for me is his response to Brewster Kahle’s question, beginning at 13:06, about why the IA does everything in-house rather tha…

What happens when there is a major earthquake in the bay area though? Not being geographically dispersed seems like a large risk.

They have backups in Canada and Egypt.
Post reply on HN