Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Internet Archive as a default host-of-record for startups

twitter.com

Post reply on HN
Print view View original

131–140 of 188 posts

Re: Internet Archive as a default host-of-record for startups

#131

Post by markjgraham »

Hi,

I manage the Wayback Machine at the Internet Archive.

Very happy so many people here care about preserving, and making available, our cultural heritage!

Please know a dedicated, and talented, team of engineers works every day to do a better job of archiving more of the public Web, and making it available via the Wayback Machine.

As noted the Internet Archive is experimenting with filecoin.io and storj.io and is always open to suggestions about how we might do our jobs better, and improve our service. We also host regular meetups (and have hosted summits and a camp) related to the Decentralized Web. See: https://blog.archive.org/tag/dweb/

The Internet Archive also offers archive-it.org, a subscription service, for those who want a higher level of support and more features.

We appreciate any support you can offer, financial and otherwise. Please share any bug reports, feature suggestions and other feedback with us via email to info@archive.org

Oh/and… checkout the new PDF Search feature we just launched at the bottom of web.archive.org. More to come like that in 2022.

Finally, you might also find some of the things I wrote here of interest: https://gijn.org/2021/05/05/tips-for-using-the-internet-arch...

Re: Internet Archive as a default host-of-record for startups

#132

Post by 1vuio0pswjnm7 »

amenghra wrote: post #127
1vuio0pswjnm7 wrote: post #124

Why is there only one IA? Why is IA not globally distributed, like a CDN? I use IA for "problem" websites, e.g., ones that rely on SNI, i.e., ones hosted at certain CDNs. I simply add add these sites to a list and the local proxy does the rest. http-request set-uri https://web.archive.org/web/1if_/http://%[req.hdr(host)]%[pathq] if { hdr(host) -m str -f list } IA "hosts" an enormous number of sites without the need f…

What’s your issue with SNI/threat model? If you use a non-SNI site, anyone can tell which site you are visiting since there’s only one domain on that IP.

There is a difference between making something "impossible" and making something "easier". Performing reverse DNS lookups, or otherwise trying to maintain a global table of 1:1 domain:IP mappings and perform lookups in real-time, is nowhere near as easy nor reliable as sniffing SNI. IME, it is neither easy nor reliable, nor worth the effort. SNI is the preferred method. SNI is easier. SNI is 100% reliable for detecting what hostname the user is trying to access.

What is the point of so-called "DNS privacy/Private DNS" if "anyone can tell which site you are visiting" simply by observing IP addresses, without any need to see domainnames.

If SNI (plaintext hostnames sent over the wire) is a non-issue, then why are people working on encrypted Client Hello in TLS1.3.

Re: Internet Archive as a default host-of-record for startups

#133

Post by thrdbndndn »

1vuio0pswjnm7 wrote: post #124

Why is there only one IA? Why is IA not globally distributed, like a CDN? I use IA for "problem" websites, e.g., ones that rely on SNI, i.e., ones hosted at certain CDNs. I simply add add these sites to a list and the local proxy does the rest. http-request set-uri https://web.archive.org/web/1if_/http://%[req.hdr(host)]%[pathq] if { hdr(host) -m str -f list } IA "hosts" an enormous number of sites without the need f…

> 1if_

This is a neat shortcut to simply get the very first archived version! I often have to go to /*/ and manually click on one of them, which is very tiring.

Is there one to get the latest?

Re: Internet Archive as a default host-of-record for startups

#135

Post by ineedasername »

zingplex wrote: post #128
ineedasername wrote: post #70

Earlier quoted context omitted.

It looks like you could use IPFS to accomplish this without using a blockchain.

Isn't IPFS pretty closely tied with FileCoin?

It seems more accurate to say filecoin is tied to ipfs. IPFS itself is just another protocol. Maybe it is better suited to Blockchain applications that https? But it doesn't require blockchain at all to function. Filecoin does require a Blockchain.

Re: Internet Archive as a default host-of-record for startups

#136

Post by [deleted] »

adventured wrote: post #76
simonw wrote: post #9

The feature I most want from the Internet Archive is the ability to donate them an old domain name and enough cash to renew it for the next hundred years such that they can keep an archived version of a site available (without breaking any incoming links) for a very long time. They would also need to be able to handle legal administration costs of things like DMCA take-down notices, but I assume they already have to…

> The feature I most want from the Internet Archive The feature I most want from IA is a streamlined system to delete content they have archived on domains that I own, including a proper privacy law compliance effort on their part. They have intentionally made it a difficult, manual process to get content removed. They operate as a de facto malicious crawler. They massively violate GDPR with how they operate and few…

[deleted]

Re: Internet Archive as a default host-of-record for startups

#137

Post by sayhar »

markjgraham wrote: post #131

Hi, I manage the Wayback Machine at the Internet Archive. Very happy so many people here care about preserving, and making available, our cultural heritage! Please know a dedicated, and talented, team of engineers works every day to do a better job of archiving more of the public Web, and making it available via the Wayback Machine. As noted the Internet Archive is experimenting with filecoin.io and storj.io and is a…

Thank you for your service!

Re: Internet Archive as a default host-of-record for startups

#138

Post by NavinF »

tablespoon wrote: post #62
toomuchtodo wrote: post #39

Earlier quoted context omitted.

It costs the Internet Archive $2/GB to host content in perpetuity. They have a tool, Archive It, that will periodically crawl your site for archival purposes if you are not technical. For my needs, I run a report monthly for the content I’ve archived using my IA account to determine archived GBs, and then donate the amount needed to cover those costs. Consider reaching out to their patron services email address with…

> It costs the Internet Archive $2/GB to host content in perpetuity. Do you have source/more info than that? Lets say the internet archive is 100 PB [1], that's 100,000,000 GB [2], and at that rate it comes out to $200 million [3] for the whole thing forever. That's a lot of money, but also a lot less than I was expecting for something like that. [1] https://www.protocol.com/internet-archive-preserving-future : "The…

Assume $11.88/TB [1], 5W per 4TB disk, and $0.2/kWh. We’re talking $1.19mm NRC + $18.3k MRC.

With a 9% discount rate, that’s only $3.63 million dollars in present value to pay for infinity months.

Of course there are other costs (and cheaper more efficient disks; and cheaper power; and your discount might be less aggressive; and server aren’t free tho you only need like 1 server for 100 disks with SAS expanders since most data is never read; and maintenance), but the $200 million number you got seems very reasonable to me.

1: https://diskprices.com/

Re: Internet Archive as a default host-of-record for startups

#139

Post by entangledqubit »

rmbyrro wrote: post #80
toomuchtodo wrote: post #39

Earlier quoted context omitted.

It costs the Internet Archive $2/GB to host content in perpetuity. They have a tool, Archive It, that will periodically crawl your site for archival purposes if you are not technical. For my needs, I run a report monthly for the content I’ve archived using my IA account to determine archived GBs, and then donate the amount needed to cover those costs. Consider reaching out to their patron services email address with…

$2/GB in perpetuity is really cheap. They should write a paper about how they did it, if they haven't already. Edit: I'm assuming they can deliver reliability and durability similar to modern cloud standards, like AWS S3.

I'd also like to see the rest of the assumptions baked in. Is there some trust fund involved -- which requires that we now consider the risk associated with how the $2 is invested to get an adequate return?

Re: Internet Archive as a default host-of-record for startups

#140

Post by chaps »

markjgraham wrote: post #131

Hi, I manage the Wayback Machine at the Internet Archive. Very happy so many people here care about preserving, and making available, our cultural heritage! Please know a dedicated, and talented, team of engineers works every day to do a better job of archiving more of the public Web, and making it available via the Wayback Machine. As noted the Internet Archive is experimenting with filecoin.io and storj.io and is a…

I worked with y'all as a volunteer back when trump was pushing to drop global warming sites from gov websites. I also use the wayback machine professionally on a regular basis. The work y'all do is genuinely appreciated to say the least.

That said.. damn I really wish y'all would revisit some of your fundamentals like recursive scraping and making sure your scraping is whole and complete before working on filecoin and other needlessly flashy systems. I'm genuinely worried y'all are digging yourselves into a technical pit that you can't get out of and it will hurt or even kill your goals.

Cheers.

Post reply on HN