Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Internet Archive as a default host-of-record for startups

twitter.com

Post reply on HN
Print view View original

151–160 of 188 posts

Re: Internet Archive as a default host-of-record for startups

#151

Post by pabs3 »

1vuio0pswjnm7 wrote: post #124

Why is there only one IA? Why is IA not globally distributed, like a CDN? I use IA for "problem" websites, e.g., ones that rely on SNI, i.e., ones hosted at certain CDNs. I simply add add these sites to a list and the local proxy does the rest. http-request set-uri https://web.archive.org/web/1if_/http://%[req.hdr(host)]%[pathq] if { hdr(host) -m str -f list } IA "hosts" an enormous number of sites without the need f…

The Tor network might be a good alternative to IA for hiding SNI (and also hiding destination IP addresses).

Re: Internet Archive as a default host-of-record for startups

#152

Post by bcherny »

chaps wrote: post #140
markjgraham wrote: post #131

Hi, I manage the Wayback Machine at the Internet Archive. Very happy so many people here care about preserving, and making available, our cultural heritage! Please know a dedicated, and talented, team of engineers works every day to do a better job of archiving more of the public Web, and making it available via the Wayback Machine. As noted the Internet Archive is experimenting with filecoin.io and storj.io and is a…

I worked with y'all as a volunteer back when trump was pushing to drop global warming sites from gov websites. I also use the wayback machine professionally on a regular basis. The work y'all do is genuinely appreciated to say the least. That said.. damn I really wish y'all would revisit some of your fundamentals like recursive scraping and making sure your scraping is whole and complete before working on filecoin an…

What’s recursive scraping (presumably, following internal links), and what’s broken about it?

Re: Internet Archive as a default host-of-record for startups

#154

Post by smarx007 »

iJohnDoe wrote: post #153

IA needs to respects robots.txt and they need to make it easier to request data to be removed. Not everyone fully supports the IA mission and they need to respect that view as much as they respect their supporters.

They clearly explained that they don’t consider archival to be a robotic activity when a person clicks “archive this page”. It’s closer to saving a page with Ctrl+S and uploading it to IA.

Removing data from the web in 2021? Hmm… https://web.archive.org/web/20211222032633/https://news.ycom... oops!

Re: Internet Archive as a default host-of-record for startups

#155

Post by jrwr »

AlexanderTheGr8 wrote: post #120

What compression does IA use to store websites? Using a 2x better compression will allow them to store 2x more websites/content. I am doing some compression research, and would love to help IA in any way I can. There are some amazing SOTA compression algorithms available now. And if IA ignores images/video, and focuses only on text, they can store an insane amount of websites at a very low cost.

right now, most of it is coming in as ZSTD with a central index. older stuff was gzip

Re: Internet Archive as a default host-of-record for startups

#156

Post by smarx007 »

chaps wrote: post #140
markjgraham wrote: post #131

Hi, I manage the Wayback Machine at the Internet Archive. Very happy so many people here care about preserving, and making available, our cultural heritage! Please know a dedicated, and talented, team of engineers works every day to do a better job of archiving more of the public Web, and making it available via the Wayback Machine. As noted the Internet Archive is experimenting with filecoin.io and storj.io and is a…

I worked with y'all as a volunteer back when trump was pushing to drop global warming sites from gov websites. I also use the wayback machine professionally on a regular basis. The work y'all do is genuinely appreciated to say the least. That said.. damn I really wish y'all would revisit some of your fundamentals like recursive scraping and making sure your scraping is whole and complete before working on filecoin an…

Recursive scraping:

1) can consume a lot of storage really fast

2) makes IA a bot rather than a user service

3) is already done on a case by case basis by ArchiveTeam allowing IA to stay away from the previous two problems.

Re: Internet Archive as a default host-of-record for startups

#157

Post by jrwr »

kodah wrote: post #3

Personally, I think eternally archiving everything and infinitely available public data has been not-so-great. If this was an "archive with consent" sort of system, then sure. My response may be better summarized as, "Does IA support robots.txt, and if not why?"

They do respect robots.txt matter of fact, it will remove the website outright and most of its archives will be hidden until that website's robots.txt is offline.

Their documentation about it is rather crap right now, but its in several of their FAQs about it

Re: Internet Archive as a default host-of-record for startups

#158

Post by i67vw3 »

markjgraham wrote: post #131

Hi, I manage the Wayback Machine at the Internet Archive. Very happy so many people here care about preserving, and making available, our cultural heritage! Please know a dedicated, and talented, team of engineers works every day to do a better job of archiving more of the public Web, and making it available via the Wayback Machine. As noted the Internet Archive is experimenting with filecoin.io and storj.io and is a…

No post body was provided.

Re: Internet Archive as a default host-of-record for startups

#159

Post by texasbigdata »

NavinF wrote: post #138
tablespoon wrote: post #62

Earlier quoted context omitted.

> It costs the Internet Archive $2/GB to host content in perpetuity. Do you have source/more info than that? Lets say the internet archive is 100 PB [1], that's 100,000,000 GB [2], and at that rate it comes out to $200 million [3] for the whole thing forever. That's a lot of money, but also a lot less than I was expecting for something like that. [1] https://www.protocol.com/internet-archive-preserving-future : "The…

Assume $11.88/TB [1], 5W per 4TB disk, and $0.2/kWh. We’re talking $1.19mm NRC + $18.3k MRC. With a 9% discount rate, that’s only $3.63 million dollars in present value to pay for infinity months. Of course there are other costs (and cheaper more efficient disks; and cheaper power; and your discount might be less aggressive; and server aren’t free tho you only need like 1 server for 100 disks with SAS expanders since…

No humans? :)

Re: Internet Archive as a default host-of-record for startups

#160

Post by iJohnDoe »

smarx007 wrote: post #154
iJohnDoe wrote: post #153

IA needs to respects robots.txt and they need to make it easier to request data to be removed. Not everyone fully supports the IA mission and they need to respect that view as much as they respect their supporters.

They clearly explained that they don’t consider archival to be a robotic activity when a person clicks “archive this page”. It’s closer to saving a page with Ctrl+S and uploading it to IA. Removing data from the web in 2021? Hmm… https://web.archive.org/web/20211222032633/https://news.ycom... oops!

Are you indicating that everything in the IA was added via a person clicking on “archive this page”?

I don’t think that is correct. A lot of it was added via automated methods?

Post reply on HN