Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Internet Archive Infrastructure

archive.org

Post reply on HN
Print view View original

61–70 of 114 posts

Re: Internet Archive Infrastructure

#61

Post by Sebb767 »

Nextgrid wrote: post #33
janjim wrote: post #28

Earlier quoted context omitted.

Curious about the cost. Does that already include manpower and various acquisition cost of constructing their internal network (hardware, fiber link between site)? I guess the biggest downside is the speed of scaling that they can do. As it is limited by how fast they can purchase and install new storage device. But with the use case of Internet Archive, that shouldn't matter much.

I've seen this argument a lot but I'm not sure how well it holds. The price for performance ratio on cloud providers is so poor that you can overprovision in advance (to mitigate the extra delay involved in adding extra hardware) and still come out ahead. Also, bare-metal doesn't necessarily mean owning the hardware. You can rent it too. There are providers that provide bare-metal in one-click and sometimes available…

> The price for performance ratio on cloud providers is so poor that you can overprovision in advance (to mitigate the extra delay involved in adding extra hardware) and still come out ahead.

It really depends on what scale you're talking about. When you're a startup and suddenly land on the front page of HN, you might need 100x or 1000x your current capacity - in which case AWS will be useful to no end.

If, on the other hand, you're an established name with quite a bit of traffic already and the maximum uptick you will reasonably experience is 2x-3x, the argument holds far less water.

Re: Internet Archive Infrastructure

#62

Post by silicon2401 »

88 wrote: post #38
lprd wrote: post #22

This is awesome! I love seeing companies run their own infrastructure. I wonder if they are using ZFS or just traditional RAID?

This is mentioned in the video, but they don’t use any form of RAID, just paired/mirrored drives in different physical locations. This is preferred for its simplicity and performance, and if I were in their position I would do the same thing.

what's the difference between RAID and paired/mirrored drives? is the latter better for NAS?

Re: Internet Archive Infrastructure

#63

Post by nodesocket »

tkgally wrote: post #5

This is a recent video presentation by Jonah Edwards, who runs the Core Infrastructure Team at the Internet Archive. He explains the IA’s server, storage, and networking infrastructure, and then takes questions from other people at the Archive. I found it all interesting. But the main takeaway for me is his response to Brewster Kahle’s question, beginning at 13:06, about why the IA does everything in-house rather tha…

What happens when there is a major earthquake in the bay area though? Not being geographically dispersed seems like a large risk.

Re: Internet Archive Infrastructure

#64

Post by 88 »

silicon2401 wrote: post #62
88 wrote: post #38

Earlier quoted context omitted.

This is mentioned in the video, but they don’t use any form of RAID, just paired/mirrored drives in different physical locations. This is preferred for its simplicity and performance, and if I were in their position I would do the same thing.

what's the difference between RAID and paired/mirrored drives? is the latter better for NAS?

RAID is an abstraction layer on top of the physical disks.

RAID 1 (mirrored drives) is similar to what the IA have described, but it sounds like they are creating their mirrors using simple file system commands (or tools like rsync) rather than introducing the complexity and overhead of hardware/software RAID.

Other forms of RAID (e.g. RAID 5/6) where data is spread across an array of drives with parity, would provide the IA with additional redundancy but at the expense of significantly increased cost and complexity.

Re: Internet Archive Infrastructure

#65

Post by shaunparker »

fireattack wrote: post #16

Probably a good place to ask: is there a reason why Wayback Machine's archives often take forever to load? Most of content on IA loads pretty fast, so WB is a notable exception.

They want you to experience what it was like to browse wayback in the early 90s. Sorry, I couldn’t resist the joke :)

Re: Internet Archive Infrastructure

#67

Post by ddorian43 »

Sebb767 wrote: post #61
Nextgrid wrote: post #33

Earlier quoted context omitted.

I've seen this argument a lot but I'm not sure how well it holds. The price for performance ratio on cloud providers is so poor that you can overprovision in advance (to mitigate the extra delay involved in adding extra hardware) and still come out ahead. Also, bare-metal doesn't necessarily mean owning the hardware. You can rent it too. There are providers that provide bare-metal in one-click and sometimes available…

> The price for performance ratio on cloud providers is so poor that you can overprovision in advance (to mitigate the extra delay involved in adding extra hardware) and still come out ahead. It really depends on what scale you're talking about. When you're a startup and suddenly land on the front page of HN, you might need 100x or 1000x your current capacity - in which case AWS will be useful to no end. If, on the o…

Feel free to point to these cases where people scaled 1000x when they hit the HN frontpage. Especially their database.

Re: Internet Archive Infrastructure

#68

Post by [deleted] »

caslon wrote: post #4
jolmg wrote: post #3

Earlier quoted context omitted.

> Wayback Machine being broken in Firefox Are you getting "Fail with status: 498 No Reason Phrase"? You might have your Referer header disabled. If that's the case, you can fix it by going to about:config and setting network.http.sendRefererHeader to 2 (or pressing the reset button to the right).

I am! Thank you! Why doesn't it work without REFERER headers? I haven't changed any settings from the Firefox default, and allowing it breaks links to jwz's site.

[deleted]

Re: Internet Archive Infrastructure

#69

Post by jmiskovic »

niea_11 wrote: post #56
jmiskovic wrote: post #49

I see incredible value in IA collection of books, videos and software. OTOH I'm puzzled by lack of organization. Take for example this newer document: https://archive.org/details/manualzilla-id-5695071 The document has horrible name and useless tags, and the content seems to be only section 2 of some SW manual. How would I ever hope to find it if I needed that exact document? Obviously such huge archive cannot be cat…

To be fair the linked document was uploaded just today to a collection that seems be considered a "waystation" collection so probably the document will be moved later to a permanent collection. And I think they have bots that process the uploaded documents to do OCR and create previews.

Even for quite mature documents, allowing users to curate them would benefit everyone.

This mature archived item https://archive.org/details/whattodrawhowtod00lutz/ is well described, but it is missing tags and it's not part of any relevant collection that would help us discover other similar books.

Re: Internet Archive Infrastructure

#70

Post by Daho0n »

owenmarshall wrote: post #55
walrus01 wrote: post #13

Earlier quoted context omitted.

On the scale of big hosting operations, 60Gbps outbound is not that much. If you're buying full tables IP transit from major carriers at IX points, I've seen 10GbE for $700-900/mo, and 100GbE circuits for under $7k/month. Of course you wouldn't want to have just one transit, but I'm fairly sure if somebody said to me 'here's $20,000 a month to buy transit', on the west coast, it's within the realm of possible. Ideall…

IME with cloud in the small and in the large: network prices are artificially high on the cloud providers and are very easy to get discounted if you are a big spender.

So the small pay for the large?
Post reply on HN