Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Internet Archive as a default host-of-record for startups

twitter.com

Post reply on HN
Print view View original

171–180 of 188 posts

Re: Internet Archive as a default host-of-record for startups

#171

Post by smarx007 »

iJohnDoe wrote: post #160
smarx007 wrote: post #154

Earlier quoted context omitted.

They clearly explained that they don’t consider archival to be a robotic activity when a person clicks “archive this page”. It’s closer to saving a page with Ctrl+S and uploading it to IA. Removing data from the web in 2021? Hmm… https://web.archive.org/web/20211222032633/https://news.ycom... oops!

Are you indicating that everything in the IA was added via a person clicking on “archive this page”? I don’t think that is correct. A lot of it was added via automated methods?

Not necessarily via a button but all of the pages were submitted to them by 3rd parties. So, they don't seem to crawl to discover new URIs eagerly, see https://news.ycombinator.com/item?id=29643506

For example, I host my own ArchiveBox at home (you get fulltext search as a bonus) and it is configured to submit every URL I save to IA: https://imgur.com/a/Yhnxo1W IA considers that to be manual submission not subject to robots.txt rules.

https://archivebox.io

Re: Internet Archive as a default host-of-record for startups

#172

Post by dreamcompiler »

gregsadetsky wrote: post #69
btrettel wrote: post #22

Earlier quoted context omitted.

A full-text search for the Wayback Machine would be my top feature request. It's not uncommon to lose the URL of a site and for active webpages to not have the URL of the old website. Plus I'm sure there are many interesting archived webpages I could find with a full-text search. I understand they've tried this or things like it a few times but they haven't ever kept the feature.

I imagine that the costs to run this would outweight the potential marketing benefits, but it'd be amazing to see Algolia take this project on to benefit everyone. The Wayback Machine's data is ~20PB..? What is approximately the size of the indexable text (i.e. the text content of html pages, sans tags)? And what would the index size be like, approximately? I imagine that creating (and maintaining, of course) the ind…

I'd think you'd want something with at least the features of Lucene, whose index is 20%-30% the size of the data.

https://lucene.apache.org/core/features.html

Re: Internet Archive as a default host-of-record for startups

#173

Post by NavinF »

texasbigdata wrote: post #159
NavinF wrote: post #138

Earlier quoted context omitted.

Assume $11.88/TB [1], 5W per 4TB disk, and $0.2/kWh. We’re talking $1.19mm NRC + $18.3k MRC. With a 9% discount rate, that’s only $3.63 million dollars in present value to pay for infinity months. Of course there are other costs (and cheaper more efficient disks; and cheaper power; and your discount might be less aggressive; and server aren’t free tho you only need like 1 server for 100 disks with SAS expanders since…

No humans? :)

Ah I was only thinking of marginal costs. I’m pretty sure the $2/GB figure is the cost of adding a little more data to the IA, not the total cost.

Re: Internet Archive as a default host-of-record for startups

#174

Post by cxr »

cookiengineer wrote: post #166
markjgraham wrote: post #131

Hi, I manage the Wayback Machine at the Internet Archive. Very happy so many people here care about preserving, and making available, our cultural heritage! Please know a dedicated, and talented, team of engineers works every day to do a better job of archiving more of the public Web, and making it available via the Wayback Machine. As noted the Internet Archive is experimenting with filecoin.io and storj.io and is a…

Does the wayback machine offer a JSON/REST API to check for URLs, in the sense of sending a URL and getting back a map of specific crawl datetimes that are available in the cache? I know about the web.archive.org/ https:// ... "hack" but it's probably draining your servers unnecessarily when there are a lot of 301 redirects that weren't archived and on top of that are only able to be validated client side after recei…

> Does the wayback machine offer a JSON/REST API to check for URLs, in the sense of sending a URL and getting back a map of specific crawl datetimes that are available in the cache?

Have you looked at the headers that they send?

  GET /web/20100330210402/https://arxiv.org/abs/0911.1112 HTTP/2
  [...]

  HTTP/2 200 OK
  [...]
  link: ; rel="original",
    ;
    rel="timemap"; type="application/link-format",
    ;
    rel="timegate",
    ;
    rel="first memento"; datetime="Tue, 30 Mar 2010 21:04:02 GMT",
    ;
    rel="memento"; datetime="Tue, 30 Mar 2010 21:04:02 GMT",
    ;
    rel="next memento"; datetime="Sat, 01 Jan 2011 20:37:56 GMT",
    ;
    rel="last memento"; datetime="Tue, 23 Nov 2021 04:06:25 GMT"
  [...]
The Wayback Machine implements RFC 7089. http://mementoweb.org/guide/quick-intro/>

Re: Internet Archive as a default host-of-record for startups

#175

Post by anaganisk »

garaetjjte wrote: post #11
kodah wrote: post #3

Personally, I think eternally archiving everything and infinitely available public data has been not-so-great. If this was an "archive with consent" sort of system, then sure. My response may be better summarized as, "Does IA support robots.txt, and if not why?"

You're consenting by posting it on public internet in the first place.

Lets say someone posted a private video and some zero day made it public, what’s the take on this scenario. Somehow robots crawled it.

Re: Internet Archive as a default host-of-record for startups

#176

Post by wolverine876 »

cookiengineer wrote: post #166
markjgraham wrote: post #131

Hi, I manage the Wayback Machine at the Internet Archive. Very happy so many people here care about preserving, and making available, our cultural heritage! Please know a dedicated, and talented, team of engineers works every day to do a better job of archiving more of the public Web, and making it available via the Wayback Machine. As noted the Internet Archive is experimenting with filecoin.io and storj.io and is a…

Does the wayback machine offer a JSON/REST API to check for URLs, in the sense of sending a URL and getting back a map of specific crawl datetimes that are available in the cache? I know about the web.archive.org/ https:// ... "hack" but it's probably draining your servers unnecessarily when there are a lot of 301 redirects that weren't archived and on top of that are only able to be validated client side after recei…

> I'm building a web browser and I'm trying to offer a feature for error cases when the server or URL isn't available anymore, so that users can see the web archived version of it.

Brave browser does that, and I've seen Firefox add-ons that do it. Maybe you can look at their code.

Re: Internet Archive as a default host-of-record for startups

#177

Post by wolverine876 »

gregsadetsky wrote: post #69
btrettel wrote: post #22

Earlier quoted context omitted.

A full-text search for the Wayback Machine would be my top feature request. It's not uncommon to lose the URL of a site and for active webpages to not have the URL of the old website. Plus I'm sure there are many interesting archived webpages I could find with a full-text search. I understand they've tried this or things like it a few times but they haven't ever kept the feature.

I imagine that the costs to run this would outweight the potential marketing benefits, but it'd be amazing to see Algolia take this project on to benefit everyone. The Wayback Machine's data is ~20PB..? What is approximately the size of the indexable text (i.e. the text content of html pages, sans tags)? And what would the index size be like, approximately? I imagine that creating (and maintaining, of course) the ind…

> Would it be enough to have an index consist of a list of found words, and the related "document ids"? i.e. "apple" is in doc ids 1000, 2000, 3000, "banana" is in doc ids 2000, 4000, etc.?

The problem, as with any search engine, is the ranking algorithm. Without a sufficent one, the search results are useless. What use is a list of every page in the Wayback Machine containing the word "apple"?

The Wayback Machine possibly would need a much larger index than any normal search engine: not only the present websites, but all the historical versions (though I don't know what proportion of the web they've indexed).

Re: Internet Archive as a default host-of-record for startups

#178

Post by wolverine876 »

boarnoah wrote: post #92
osigurdson wrote: post #34

Earlier quoted context omitted.

Perhaps have a look at Arweave ( https://www.arweave.org ).

That doesn't address the concern re: something needing critical mass to increase its chance of survival over a longer term. Really the way I see it outside a few large banking firms, its kind of hard to be sure any provider of digital services would be around in the 50+ year term for this kind of public archive. I hope the Internet Archive manages it. EDIT: I do worry the IA has a bit of a lightning rod effect with s…

> Really the way I see it outside a few large banking firms, its kind of hard to be sure any provider of digital services would be around in the 50+ year term for this kind of public archive.

A library could do it. Perhaps leading institutions like the British Library or Library of Congress. I've thought that IA should be a Library of Congress project, and may eventually end up under their auspices.

Re: Internet Archive as a default host-of-record for startups

#179

Post by gregsadetsky »

wolverine876 wrote: post #177
gregsadetsky wrote: post #69

Earlier quoted context omitted.

I imagine that the costs to run this would outweight the potential marketing benefits, but it'd be amazing to see Algolia take this project on to benefit everyone. The Wayback Machine's data is ~20PB..? What is approximately the size of the indexable text (i.e. the text content of html pages, sans tags)? And what would the index size be like, approximately? I imagine that creating (and maintaining, of course) the ind…

> Would it be enough to have an index consist of a list of found words, and the related "document ids"? i.e. "apple" is in doc ids 1000, 2000, 3000, "banana" is in doc ids 2000, 4000, etc.? The problem, as with any search engine, is the ranking algorithm. Without a sufficent one, the search results are useless. What use is a list of every page in the Wayback Machine containing the word "apple"? The Wayback Machine po…

No you are right of course. I was imagining that a full text Wayback Machine search engine would mostly be useful to look for words unique enough that sifting through a lot of results (even if those were not ranked "well") could still be useful..?

I was very naively "back of the envelope" prototyping a search engine. I both realize that this is not the way that these things are built, and both would really like to have that (or any) search engine to look through the archive..! :-)

As for the index, I agree -- I was trying to guesstimate its size by going from the ~~20PB total Wayback Machine size (which includes all historical versions). Is it 1% of 20PB (for the size of the text content), and then another ~10% of that for the index size? So 20TB...?

Re: Internet Archive as a default host-of-record for startups

#180

Post by wolverine876 »

gregsadetsky wrote: post #179
wolverine876 wrote: post #177

Earlier quoted context omitted.

> Would it be enough to have an index consist of a list of found words, and the related "document ids"? i.e. "apple" is in doc ids 1000, 2000, 3000, "banana" is in doc ids 2000, 4000, etc.? The problem, as with any search engine, is the ranking algorithm. Without a sufficent one, the search results are useless. What use is a list of every page in the Wayback Machine containing the word "apple"? The Wayback Machine po…

No you are right of course. I was imagining that a full text Wayback Machine search engine would mostly be useful to look for words unique enough that sifting through a lot of results (even if those were not ranked "well") could still be useful..? I was very naively "back of the envelope" prototyping a search engine. I both realize that this is not the way that these things are built, and both would really like to ha…

I may have given the wrong impression: it sounds like a good idea to me.

> I was imagining that a full text Wayback Machine search engine would mostly be useful to look for words unique enough that sifting through a lot of results (even if those were not ranked "well") could still be useful..?

If we think about use cases, users may often search specific domains. In that case, results ordered by frequency and/or date might be sufficient and even desireable.

> I was very naively "back of the envelope" prototyping a search engine. I both realize that this is not the way that these things are built

It's often the first step!

> So 20TB...?

That doesn't sound so bad.

Post reply on HN