Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Internet Archive Infrastructure

archive.org

Post reply on HN
Print view View original

71–80 of 114 posts

Re: Internet Archive Infrastructure

#71

Post by xhrpost »

walrus01 wrote: post #13
benaadams wrote: post #10

Earlier quoted context omitted.

I assume the 200PB of storage and 60Gbps egress bandwidth 24/365 they do would be _extremely_ pricy on AWS...

On the scale of big hosting operations, 60Gbps outbound is not that much. If you're buying full tables IP transit from major carriers at IX points, I've seen 10GbE for $700-900/mo, and 100GbE circuits for under $7k/month. Of course you wouldn't want to have just one transit, but I'm fairly sure if somebody said to me 'here's $20,000 a month to buy transit', on the west coast, it's within the realm of possible. Ideall…

It seems they are regularly maxing out their network infrastructure. If it's so cheap, how come they don't just buy more? Is it the cost of the actual hardware? (I know they recently upgraded)

Re: Internet Archive Infrastructure

#72

Post by jeromegv »

Daho0n wrote: post #70
owenmarshall wrote: post #55

Earlier quoted context omitted.

IME with cloud in the small and in the large: network prices are artificially high on the cloud providers and are very easy to get discounted if you are a big spender.

So the small pay for the large?

Yes, that's typically how it works for any industry. You get discount on bulk orders.

Re: Internet Archive Infrastructure

#73

Post by Ansil849 »

tkgally wrote: post #5

This is a recent video presentation by Jonah Edwards, who runs the Core Infrastructure Team at the Internet Archive. He explains the IA’s server, storage, and networking infrastructure, and then takes questions from other people at the Archive. I found it all interesting. But the main takeaway for me is his response to Brewster Kahle’s question, beginning at 13:06, about why the IA does everything in-house rather tha…

> and greater confidence that their users are not being tracked.

And that right there is why I continue to donate to IA. I am sick and tired of services offloading my data to destructive companies like Amazon.

Re: Internet Archive Infrastructure

#74

Post by rsync »

walrus01 wrote: post #13
benaadams wrote: post #10

Earlier quoted context omitted.

I assume the 200PB of storage and 60Gbps egress bandwidth 24/365 they do would be _extremely_ pricy on AWS...

On the scale of big hosting operations, 60Gbps outbound is not that much. If you're buying full tables IP transit from major carriers at IX points, I've seen 10GbE for $700-900/mo, and 100GbE circuits for under $7k/month. Of course you wouldn't want to have just one transit, but I'm fairly sure if somebody said to me 'here's $20,000 a month to buy transit', on the west coast, it's within the realm of possible. Ideall…

Two words: hurricane electric.

Re: Internet Archive Infrastructure

#75

Post by JKCalhoun »

101008 wrote: post #46

When I was a kid, my dreams was to work at Google, Microsoft, Apple or any of these big companies. Now that I am reaching 30 (and becoming very nostlagic of the old web), I think the company that would make me most happy of login every morning to get some work done would be Internet Archive.

I love archive.org.

Hate their "front end". One of the most disorganized user-facing sites.

I would love to see an effort to address that.

Would it be possible for archive.org to offer an API to allow other sites to present archive.org using their own front-end? We could see lots of specialty sites that focus on the user experience for some slice of archive.org.

Re: Internet Archive Infrastructure

#76

Post by JKCalhoun »

jmiskovic wrote: post #49

I see incredible value in IA collection of books, videos and software. OTOH I'm puzzled by lack of organization. Take for example this newer document: https://archive.org/details/manualzilla-id-5695071 The document has horrible name and useless tags, and the content seems to be only section 2 of some SW manual. How would I ever hope to find it if I needed that exact document? Obviously such huge archive cannot be cat…

Commented above before I saw yours. I agree, and wonder further if archive.org could play host instead to any number of spinoff sites that try to better organize/present the data (or a subset of the data) on archive.org.

Re: Internet Archive Infrastructure

#77

Post by ExtraE »

jraph wrote: post #7
jolmg wrote: post #6

Earlier quoted context omitted.

> Why doesn't it work without REFERER headers? Don't know. I just spotted that it's the only meaningful difference I had in my traffic between visiting with Firefox and with Chromium. IIRC, it's the fetching of the capture timestamps that fails when the Referer header is missing. > I haven't changed any settings from the Firefox default You might have forgotten like I did once. > and allowing it breaks links to jwz's…

It won't if you click a jwz link from Hacker News. It shows you a testicle in an egg cup. This is on purpose.

How does this work? I was under the impression that when I clicked a link in a webpage it just opened that link as if I'd typed it into my URL bar. How does jwz know were I'm clicking from?

Re: Internet Archive Infrastructure

#80

Post by dweekly »

rsync wrote: post #74
walrus01 wrote: post #13

Earlier quoted context omitted.

On the scale of big hosting operations, 60Gbps outbound is not that much. If you're buying full tables IP transit from major carriers at IX points, I've seen 10GbE for $700-900/mo, and 100GbE circuits for under $7k/month. Of course you wouldn't want to have just one transit, but I'm fairly sure if somebody said to me 'here's $20,000 a month to buy transit', on the west coast, it's within the realm of possible. Ideall…

Two words: hurricane electric.

God bless HE Fremont. They are the unsung story of the Internet backbone. If one were to make a list of companies that at some point had a major fraction of their hosted physical infrastructure at HE I suspect it would make people's jaws drop.

It's been a huge blessing for a whole generation of startups to have a radically well-connected space that just about anyone can drop in equipment at multi-gig unmetered (albeit admittedly extremely constrained on power and cooling). It is honestly a part of what has made Silicon Valley great. Well, that and being able to cobble together a few replacement servers from Fry's components (RIP!) or schlep out to some ex-CTO's Sunnyvale garage in the middle of the night to offload some lightly used VA Linux 1U's...

Even today, you can get 15A and a 42U cabinet to call your own with unmetered gigabit for $400/mo - and probably less if you ask nicely.

Post reply on HN