Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Internet Archive as a default host-of-record for startups

twitter.com

Post reply on HN
Print view View original

161–170 of 188 posts

Re: Internet Archive as a default host-of-record for startups

#161

Post by mysterydip »

markjgraham wrote: post #131

Hi, I manage the Wayback Machine at the Internet Archive. Very happy so many people here care about preserving, and making available, our cultural heritage! Please know a dedicated, and talented, team of engineers works every day to do a better job of archiving more of the public Web, and making it available via the Wayback Machine. As noted the Internet Archive is experimenting with filecoin.io and storj.io and is a…

Just wanted to say thank you for all the times you've let me continue researching when the original sources became dead!

Re: Internet Archive as a default host-of-record for startups

#162

Post by eslaught »

smarx007 wrote: post #156
chaps wrote: post #140

Earlier quoted context omitted.

I worked with y'all as a volunteer back when trump was pushing to drop global warming sites from gov websites. I also use the wayback machine professionally on a regular basis. The work y'all do is genuinely appreciated to say the least. That said.. damn I really wish y'all would revisit some of your fundamentals like recursive scraping and making sure your scraping is whole and complete before working on filecoin an…

Recursive scraping: 1) can consume a lot of storage really fast 2) makes IA a bot rather than a user service 3) is already done on a case by case basis by ArchiveTeam allowing IA to stay away from the previous two problems.

There's an old school website I wanted to access that's long gone. It's only partially preserved in the Wayback Machine. I'm glad it's there at all, but some of the specific information I was looking for was never captured, and now there's no way to recover it.

I realize it's a hard problem, but I really wish there were a way to automate more of it. Some of these communities are too small for anyone to bother preserving the pages manually, and I don't imagine we'd even show up on ArchiveTeam's radar. But they're not large pages, and basically static. I don't think they'd be a huge burden to store and maintain. It seems like some sort of a coverage + size metric would be pretty effective at guiding an automated scan such that you'd be able to preserve things liked this without needing humans to go and manually archive each and every page.

Re: Internet Archive as a default host-of-record for startups

#163

Post by chaps »

eslaught wrote: post #162
smarx007 wrote: post #156

Earlier quoted context omitted.

Recursive scraping: 1) can consume a lot of storage really fast 2) makes IA a bot rather than a user service 3) is already done on a case by case basis by ArchiveTeam allowing IA to stay away from the previous two problems.

There's an old school website I wanted to access that's long gone. It's only partially preserved in the Wayback Machine. I'm glad it's there at all, but some of the specific information I was looking for was never captured, and now there's no way to recover it. I realize it's a hard problem, but I really wish there were a way to automate more of it. Some of these communities are too small for anyone to bother preserv…

Pretty much. Your idea of getting coverage could probably work well to make sure IA doesn't "turn into a bot" (though it's practically the same, just without the permanent downloading).

Re: Internet Archive as a default host-of-record for startups

#164

Post by ivank »

eslaught wrote: post #162
smarx007 wrote: post #156

Earlier quoted context omitted.

Recursive scraping: 1) can consume a lot of storage really fast 2) makes IA a bot rather than a user service 3) is already done on a case by case basis by ArchiveTeam allowing IA to stay away from the previous two problems.

There's an old school website I wanted to access that's long gone. It's only partially preserved in the Wayback Machine. I'm glad it's there at all, but some of the specific information I was looking for was never captured, and now there's no way to recover it. I realize it's a hard problem, but I really wish there were a way to automate more of it. Some of these communities are too small for anyone to bother preserv…

https://wiki.archiveteam.org/index.php/ArchiveBot is available if you would like to help save things that are still up.

Re: Internet Archive as a default host-of-record for startups

#165

Post by chaps »

bcherny wrote: post #152
chaps wrote: post #140

Earlier quoted context omitted.

I worked with y'all as a volunteer back when trump was pushing to drop global warming sites from gov websites. I also use the wayback machine professionally on a regular basis. The work y'all do is genuinely appreciated to say the least. That said.. damn I really wish y'all would revisit some of your fundamentals like recursive scraping and making sure your scraping is whole and complete before working on filecoin an…

What’s recursive scraping (presumably, following internal links), and what’s broken about it?

Yep, that's exactly it. Recursive scraping, in theory, would remove a lot gaps on IA. It's really hard to do generically and "right" though. Lots of one-off.. strangeness.. especially from older sites and sites that auto-generate URLs at render time.

Re: Internet Archive as a default host-of-record for startups

#166

Post by cookiengineer »

markjgraham wrote: post #131

Hi, I manage the Wayback Machine at the Internet Archive. Very happy so many people here care about preserving, and making available, our cultural heritage! Please know a dedicated, and talented, team of engineers works every day to do a better job of archiving more of the public Web, and making it available via the Wayback Machine. As noted the Internet Archive is experimenting with filecoin.io and storj.io and is a…

Does the wayback machine offer a JSON/REST API to check for URLs, in the sense of sending a URL and getting back a map of specific crawl datetimes that are available in the cache?

I know about the web.archive.org/https://... "hack" but it's probably draining your servers unnecessarily when there are a lot of 301 redirects that weren't archived and on top of that are only able to be validated client side after receiving the whole HTML response.

A REST API would help thirdparty clients to know about this in advance, and the API documentations I found were super unclear in whether something like this exists or not.

Context: I'm building a web browser and I'm trying to offer a feature for error cases when the server or URL isn't available anymore, so that users can see the web archived version of it.

On top of that I have no idea how to "un-UI" the web archived versions. I know that the wget user agent somehow leads to this, but it's also kind of undocumented how the webserver of IA does this in the background and when exactly the UI is injected and all the URLs are rewritten. Something like maybe a http request header to get the raw actual source would be nice.

Re: Internet Archive as a default host-of-record for startups

#167

Post by akiselev »

entangledqubit wrote: post #139
rmbyrro wrote: post #80

Earlier quoted context omitted.

$2/GB in perpetuity is really cheap. They should write a paper about how they did it, if they haven't already. Edit: I'm assuming they can deliver reliability and durability similar to modern cloud standards, like AWS S3.

I'd also like to see the rest of the assumptions baked in. Is there some trust fund involved -- which requires that we now consider the risk associated with how the $2 is invested to get an adequate return?

It's based on the assumption that storage costs per GB will continue to decrease for their definition of "perpetuity." If you assume a consistent fall in costs, the total cost becomes a convergent series [1] and there is an upper limit even with an infinite number of years.

Suffice it to say, physics is probably going to have a lot to say about that assumption in the coming decades.

[1] https://en.wikipedia.org/wiki/Convergent_series

Re: Internet Archive as a default host-of-record for startups

#168

Post by eitland »

boarnoah wrote: post #100
bananamerica wrote: post #68

Earlier quoted context omitted.

Hard to know what will be of interest for future historians. Some things in which we place great value can be considered irrelevant, while some of our junk can become historical gold.

The mundane of today is very insightful for tomorrow's historians. Its fascinating when you start looking into any historical time period (you wouldn't even need to go far back), before a lot of details are educated guesses. Since no one chose to record the mundane in detail or it failed to preserve over time.

> The mundane of today is very insightful for tomorrow's historians.

Too add an example I know well.

I grew up on a small farm.

We have plenty of images of Christmas parties etc, but almost none showing actual work being done which is what I think my kids would appreciate the most.

Luckily YouTube for all its warts exist and I can look up the "motorized tea spoon", the U-9 Motostandard for them when I need to explain it: https://www.youtube.com/results?search_query=motostandard+u9...

(We had the one with front-mounted wagon and a stick for steering. And yes we had another slightly larger tractor as well, the AEBI Transporter TP50: )

Re: Internet Archive as a default host-of-record for startups

#169

Post by LegionMammal978 »

cookiengineer wrote: post #166
markjgraham wrote: post #131

Hi, I manage the Wayback Machine at the Internet Archive. Very happy so many people here care about preserving, and making available, our cultural heritage! Please know a dedicated, and talented, team of engineers works every day to do a better job of archiving more of the public Web, and making it available via the Wayback Machine. As noted the Internet Archive is experimenting with filecoin.io and storj.io and is a…

Does the wayback machine offer a JSON/REST API to check for URLs, in the sense of sending a URL and getting back a map of specific crawl datetimes that are available in the cache? I know about the web.archive.org/ https:// ... "hack" but it's probably draining your servers unnecessarily when there are a lot of 301 redirects that weren't archived and on top of that are only able to be validated client side after recei…

Regarding your last point, adding "id_" to the end of the timestamp in the URL produces the original downloaded file. Adding "if_" produces the page with rewritten links but without the Wayback Machine header. Additionally, if you look at the HTTP response headers on any version of the archived page, they include copies of the original response headers.

Re: Internet Archive as a default host-of-record for startups

#170

Post by smarx007 »

eslaught wrote: post #162
smarx007 wrote: post #156

Earlier quoted context omitted.

Recursive scraping: 1) can consume a lot of storage really fast 2) makes IA a bot rather than a user service 3) is already done on a case by case basis by ArchiveTeam allowing IA to stay away from the previous two problems.

There's an old school website I wanted to access that's long gone. It's only partially preserved in the Wayback Machine. I'm glad it's there at all, but some of the specific information I was looking for was never captured, and now there's no way to recover it. I realize it's a hard problem, but I really wish there were a way to automate more of it. Some of these communities are too small for anyone to bother preserv…

I never got turned down sending an IRC message "x is going down in a few days" to the AT channels. https://wiki.archiveteam.org/index.php/Archiveteam:IRC

AT just asks to be given a few weeks/months of notice depending on how many TBs (MBs?) the crawl needs to be and not to throttle/ban their clients.

Post reply on HN