Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Internet Archive as a default host-of-record for startups

twitter.com

Post reply on HN
Print view View original

141–150 of 188 posts

Re: Internet Archive as a default host-of-record for startups

#141

Post by Ericson2314 »

kodah wrote: post #111
Ericson2314 wrote: post #79

Earlier quoted context omitted.

There is no reason to do person communication with staticish websites. Don't foist the problems of social media onto the Internet Archive.

I'm not sure I understand. robots.txt was a matter of consent and was standardized well over twenty years ago. The IA willfully ignores it because they believe it interferes with their mission. This was long before social media, when static websites were more dominant than dynamic ones. Source: https://blog.archive.org/2017/04/17/robots-txt-meant-for-sea...

This is a fair philosophical quibble, but the tiny amount of harm done in in real world makes this a waste of time discussing.

You made it public on the damn web, people doing what they wanted with it is fair game per the original ethos of the web. If you wanted it to be private, you should have some auth or encryption.

The actual real world harm is in actually-common-used means of personal communication were network effects preclude on carrying out they business with technology appropriate to desired privacy levels. Chatting on my custom website was always niche, and therefore the network effects argument doesn't carry water.

We should refrain from worrying about robots.txt minute until the elephant in the room is put to rset.

Re: Internet Archive as a default host-of-record for startups

#142

Post by kodah »

Ericson2314 wrote: post #141
kodah wrote: post #111

Earlier quoted context omitted.

I'm not sure I understand. robots.txt was a matter of consent and was standardized well over twenty years ago. The IA willfully ignores it because they believe it interferes with their mission. This was long before social media, when static websites were more dominant than dynamic ones. Source: https://blog.archive.org/2017/04/17/robots-txt-meant-for-sea...

This is a fair philosophical quibble, but the tiny amount of harm done in in real world makes this a waste of time discussing. You made it public on the damn web, people doing what they wanted with it is fair game per the original ethos of the web. If you wanted it to be private, you should have some auth or encryption. The actual real world harm is in actually-common-used means of personal communication were network…

Issues are not single track. I can rightfully claim that IA is founded on unethical behavior while I can also say other things in the world are bad.

Re: Internet Archive as a default host-of-record for startups

#144

Post by BenjiWiebe »

kodah wrote: post #109
1MachineElf wrote: post #5

Earlier quoted context omitted.

There is some public discussion about why IA does not strictly adhere to robots.txt: https://blog.archive.org/2017/04/17/robots-txt-meant-for-sea...

They're basically saying they're choosing to ignore a web convention that explicitly states that people don't want their websites archived or searchable because they want them to be. Sounds pretty unethical to me.

When those people die and quit paying for hosting, the information on their website doesn't magically become useless.

Perhaps other people still have a need for it.

Thank goodness the IA doesn't blindly obey robots.txt.

Re: Internet Archive as a default host-of-record for startups

#145

Post by 1vuio0pswjnm7 »

thrdbndndn wrote: post #133
1vuio0pswjnm7 wrote: post #124

Why is there only one IA? Why is IA not globally distributed, like a CDN? I use IA for "problem" websites, e.g., ones that rely on SNI, i.e., ones hosted at certain CDNs. I simply add add these sites to a list and the local proxy does the rest. http-request set-uri https://web.archive.org/web/1if_/http://%[req.hdr(host)]%[pathq] if { hdr(host) -m str -f list } IA "hosts" an enormous number of sites without the need f…

> 1if_ This is a neat shortcut to simply get the very first archived version! I often have to go to /*/ and manually click on one of them, which is very tiring. Is there one to get the latest?

There might be, although I would not be surprised if it was slower than the one for the link to the first because the link to the latest is dynamic. A two-step "shortcut" is to use Lua with the proxy to retrieve the link to the latest then follow that link.

To get the link to the latest, can use memento. For example,

   usage: echo example.com|1.sh

   #! /bin/sh
   read x;
   curl -A "" https://web.archive.org/web/$(curl -A "" -s "https://web.archive.org/cdx/search/cdx?url=$x&fl=timestamp,original&limit=-1"|tr \\40 /)

Re: Internet Archive as a default host-of-record for startups

#146

Post by dbmikus »

ineedasername wrote: post #55

I don't understand why Carmack thinks blockchain should be a component of this. Anyone care to elaborate on how that would make this easier/better?

Filecoin is a mechanism for someone to pay to ensure that data stays available. So if a group of people wanted to ensure that myoldwebsite.com stays available through IPFS and the IA, they could spend Filecoin. The difference between it and paying some central provider with fiat is that anyone can provide the data availability. So if you don't trust IA to be the long-term provider of content or want a more decentralized provider, Filecoin lets you do that. See also Arweave.

Re: Internet Archive as a default host-of-record for startups

#147

Post by aksss »

toomuchtodo wrote: post #85
rmbyrro wrote: post #80

Earlier quoted context omitted.

$2/GB in perpetuity is really cheap. They should write a paper about how they did it, if they haven't already. Edit: I'm assuming they can deliver reliability and durability similar to modern cloud standards, like AWS S3.

Self built storage nodes and software for storage system, with an expectation that storage costs continue to decline per GB into the future.

beowolf cluster

Re: Internet Archive as a default host-of-record for startups

#148

Post by aksss »

entangledqubit wrote: post #139
rmbyrro wrote: post #80

Earlier quoted context omitted.

$2/GB in perpetuity is really cheap. They should write a paper about how they did it, if they haven't already. Edit: I'm assuming they can deliver reliability and durability similar to modern cloud standards, like AWS S3.

I'd also like to see the rest of the assumptions baked in. Is there some trust fund involved -- which requires that we now consider the risk associated with how the $2 is invested to get an adequate return?

I too am a bit confused about that $2/GB for perpetuity. I'm assuming that's a loaded cost representing maintenance, backups, power. A TCO kind of number. Maybe that's a bad assumption?

Re: Internet Archive as a default host-of-record for startups

#149

Post by gjs278 »

chaps wrote: post #140
markjgraham wrote: post #131

Hi, I manage the Wayback Machine at the Internet Archive. Very happy so many people here care about preserving, and making available, our cultural heritage! Please know a dedicated, and talented, team of engineers works every day to do a better job of archiving more of the public Web, and making it available via the Wayback Machine. As noted the Internet Archive is experimenting with filecoin.io and storj.io and is a…

I worked with y'all as a volunteer back when trump was pushing to drop global warming sites from gov websites. I also use the wayback machine professionally on a regular basis. The work y'all do is genuinely appreciated to say the least. That said.. damn I really wish y'all would revisit some of your fundamentals like recursive scraping and making sure your scraping is whole and complete before working on filecoin an…

No post body was provided.

Re: Internet Archive as a default host-of-record for startups

#150

Post by gjs278 »

rmbyrro wrote: post #80
toomuchtodo wrote: post #39

Earlier quoted context omitted.

It costs the Internet Archive $2/GB to host content in perpetuity. They have a tool, Archive It, that will periodically crawl your site for archival purposes if you are not technical. For my needs, I run a report monthly for the content I’ve archived using my IA account to determine archived GBs, and then donate the amount needed to cover those costs. Consider reaching out to their patron services email address with…

$2/GB in perpetuity is really cheap. They should write a paper about how they did it, if they haven't already. Edit: I'm assuming they can deliver reliability and durability similar to modern cloud standards, like AWS S3.

No post body was provided.
Post reply on HN