Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Internet Archive Infrastructure

archive.org

Post reply on HN
Print view View original

81–90 of 114 posts

Re: Internet Archive Infrastructure

#81

Post by blacksmith_tb »

polote wrote: post #44
charcircuit wrote: post #12

Earlier quoted context omitted.

I have no clue how people can afford what AWS charges for bandwidth. I did the math once for migrating a project to AWS and the bandwidth alone costed 10x my entire current infrastructure for that project, which is something I run for free.

Because people have nothing to compare their AWS cost to. They don't know how much it would cost them to host their service outside of AWS. And it is not only a cost comparison. You need different kind of people to manage in-house vs cloud, not that you need less or more, just different skills

Absolutely - AWS is this generation's IBM, no one was ever fired for buying services from them, so to speak.

Re: Internet Archive Infrastructure

#82

Post by samstave »

Ansil849 wrote: post #73
tkgally wrote: post #5

This is a recent video presentation by Jonah Edwards, who runs the Core Infrastructure Team at the Internet Archive. He explains the IA’s server, storage, and networking infrastructure, and then takes questions from other people at the Archive. I found it all interesting. But the main takeaway for me is his response to Brewster Kahle’s question, beginning at 13:06, about why the IA does everything in-house rather tha…

> and greater confidence that their users are not being tracked. And that right there is why I continue to donate to IA. I am sick and tired of services offloading my data to destructive companies like Amazon.

Tracking should be STRICTLY illegal and ONLY acceptable with a verifiable OPT-IN and a transparency on to WHOM the data was sent/read-by/received. With STILL an option to selectively opt-out.

Re: Internet Archive Infrastructure

#83

Post by niea_11 »

jmiskovic wrote: post #69
niea_11 wrote: post #56

Earlier quoted context omitted.

To be fair the linked document was uploaded just today to a collection that seems be considered a "waystation" collection so probably the document will be moved later to a permanent collection. And I think they have bots that process the uploaded documents to do OCR and create previews.

Even for quite mature documents, allowing users to curate them would benefit everyone. This mature archived item https://archive.org/details/whattodrawhowtod00lutz/ is well described, but it is missing tags and it's not part of any relevant collection that would help us discover other similar books.

I agree that getting more people involved (by crowdsourcing or other ways) will be beneficial to the project. But it'll depend on how the effort will be organized.

My experience with the website is that the original uploader can edit some part of the "metadata" (not all) of the uploaded document like title, description, topics/tags ..., but they can't move it to a different collection (initially the document is uploaded the "community" collection).

If they want to put it in a different collection they have to contact archive.org's staff.

Sometimes the staff notice the new files and move them to the correct collection or even create a new one for them (the latter case happened to me).

Re: Internet Archive Infrastructure

#84

Post by Shared404 »

samstave wrote: post #82
Ansil849 wrote: post #73

Earlier quoted context omitted.

> and greater confidence that their users are not being tracked. And that right there is why I continue to donate to IA. I am sick and tired of services offloading my data to destructive companies like Amazon.

Tracking should be STRICTLY illegal and ONLY acceptable with a verifiable OPT-IN and a transparency on to WHOM the data was sent/read-by/received. With STILL an option to selectively opt-out.

I agree with the sentiment, but there needs to be some degree of leeway.

For example are server logs considered tracking? It seems unreasonable to require that logs not be kept.

Edit: On further thought, I don't even know that tracking should be banned. Instead, I would argue that advertising in the way enabled by tracking should be banned. That way the incentive is removed with less bureaucracy.

Re: Internet Archive Infrastructure

#85

Post by jolmg »

ExtraE wrote: post #77
jraph wrote: post #7

Earlier quoted context omitted.

It won't if you click a jwz link from Hacker News. It shows you a testicle in an egg cup. This is on purpose.

How does this work? I was under the impression that when I clicked a link in a webpage it just opened that link as if I'd typed it into my URL bar. How does jwz know were I'm clicking from?

https://en.wikipedia.org/wiki/HTTP_referer

> The HTTP referer (a misspelling of referrer) is an optional HTTP header field that identifies the address of the webpage (i.e., the URI or IRI), which is linked to the resource being requested. By checking the referrer, the new webpage can see where the request originated.

https://tools.ietf.org/html/rfc7231#page-45

> The Referer header field allows servers to generate back-links to other resources for simple analytics, logging, optimized caching, etc. It also allows obsolete or mistyped links to be found for maintenance. Some servers use the Referer header field as a means of denying links from other sites (so-called "deep linking") or restricting cross-site request forgery (CSRF), but not all requests contain it.

Re: Internet Archive Infrastructure

#86

Post by ant6n »

JKCalhoun wrote: post #76
jmiskovic wrote: post #49

I see incredible value in IA collection of books, videos and software. OTOH I'm puzzled by lack of organization. Take for example this newer document: https://archive.org/details/manualzilla-id-5695071 The document has horrible name and useless tags, and the content seems to be only section 2 of some SW manual. How would I ever hope to find it if I needed that exact document? Obviously such huge archive cannot be cat…

Commented above before I saw yours. I agree, and wonder further if archive.org could play host instead to any number of spinoff sites that try to better organize/present the data (or a subset of the data) on archive.org.

It would be nice if there was a good search engine for the wayback engine. Browsing in the past is pretty cumbersome right now, you need to know the URLs or at least websites that used to have the information you want.

Re: Internet Archive Infrastructure

#87

Post by carapace »

coldpie wrote: post #58
beckman466 wrote: post #35

Earlier quoted context omitted.

I'm surprised to read this and OP's comment, for me it's always quite fast. Are there specific websites your request - are they media heavy?

I picked a random article from my browser's history, linked below. Just loading the snapshot year pages took about 10 seconds, then the snapshot hover took another 10 seconds. Finally, fully loading the snapshot page took about 50 seconds. So that's roughly 90 seconds to go from inserting a URL into the search bar and actually having the rendered page. Not unacceptable, but certainly slow by modern standards. http://…

90s is better than ∞ (infinity) which is what it would be if the Archive didn't exist, eh?

Re: Internet Archive Infrastructure

#88

Post by bnewbold »

janjim wrote: post #28
tkgally wrote: post #5

This is a recent video presentation by Jonah Edwards, who runs the Core Infrastructure Team at the Internet Archive. He explains the IA’s server, storage, and networking infrastructure, and then takes questions from other people at the Archive. I found it all interesting. But the main takeaway for me is his response to Brewster Kahle’s question, beginning at 13:06, about why the IA does everything in-house rather tha…

Curious about the cost. Does that already include manpower and various acquisition cost of constructing their internal network (hardware, fiber link between site)? I guess the biggest downside is the speed of scaling that they can do. As it is limited by how fast they can purchase and install new storage device. But with the use case of Internet Archive, that shouldn't matter much.

If you are interested in cost modeling for long-term digital preservation, check out this blog series: https://blog.dshr.org/2019/02/economic-models-of-long-term-s...

Re: Internet Archive Infrastructure

#89

Post by bnewbold »

npunt wrote: post #17
fireattack wrote: post #16

Probably a good place to ask: is there a reason why Wayback Machine's archives often take forever to load? Most of content on IA loads pretty fast, so WB is a notable exception.

I get the impression Wayback Machine data is stored in powered down drives and they only spin them up when someone accesses the data. That would explain the several second delay and it'd make sense that an archive wouldn't need 95% of its data ready to go at a moments notice since that'd be a terrible waste of power.

The disks are spinning all the time, and most disks are seeing fairly frequent reads to some content or another. A lot of content is very rarely accesses, but almost every disk has some content which gets accessed. If spinning disks had only frequently-accessed content, they would be unable to keep up with the read rate or read throughput, things balance out reasonably on average.

Wayback content is on the same disks as most other content, in the form of WARC files, with individual records fetched out of the middle of WARC files via HTTP range request.

Note: I work at IA but am not on core infrastructure team

Re: Internet Archive Infrastructure

#90

Post by bnewbold »

fireattack wrote: post #16

Probably a good place to ask: is there a reason why Wayback Machine's archives often take forever to load? Most of content on IA loads pretty fast, so WB is a notable exception.

Performance is fun!

One aspect is that our data centers are in California, with no CDN. If you are on the other side of the world, you will have higher round-trip latency on every request, for all services.

Another is layers of caching. Popular or recently requested Wayback content is more likely to be in either an explicit cache (eg, redis), or implicitly in kernel page caches across all layers of the request.

Every wayback replay request hits several layers of index indexes (sorted by domain, path, and timestamp), which are huge and thus actually served from spinning disk over HTTP (!). This includes a timeline summary for the primary document, to display the banner. Then the actual raw records are fetched from another spinning disk over HTTP. This may result in one or more layers of internal redirect (additional fetches) if there was a "revisit" (identical HTTP body content, same URL, different timestamp). Then finally the record is re-written for replay (for HTML, CSS, Javascript, etc, unless the raw record was requested). Some pages will have many sub-resources, so this process is repeated many times, but that is the same as page load and you can see which resources are slow or not.

As mentioned in the video, depending on where we are in the network hardware upgrade lifecycle, sometimes outbound bandwidth is tight also, which slows down transfer.

And of course most of these services operate without a ton of overhead, so if there is a spike in traffic everything will slow down a bit. There is a lot of multi-tenancy-like situations also, so if there is a very popular zip file or Flash game getting served from the same storage disk as the WARC file holding a wayback resource, the replay for that specific resource will be slow due to disk I/O contention.

If you are curious about why a specific HTML wayback replay was slow, you can look in the source code of the re-written document and see some timing numbers.

Several organizations run large web archives that operate similarly to web.archive.org, and have described cost/benefit trade offs for different components. Eg, National Library of Australia has an alternative CDX index called OutbackCDX, which uses RocksDB on SSDs. I believe other folks store WARC files in S3 or S3-like object storage systems. The Wayback Machine is somewhat unique in the amount of (read) traffic it gets, the heterogeneity of archived content (from several crawlers, in older ARC as well as WARC), volume of live crawling ("save paper now" results show up pretty fast in the main site, which is black magic), running on "boring" general purpose hardware, and deep integration with our general purpose storage cluster.

Note: I work at IA but not on the Wayback system

Post reply on HN