Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Tesoro – Personal internet archive

tesoro.io

Post reply on HN
Print view View original

81–90 of 105 posts

Re: Show HN: Tesoro – Personal internet archive

#81

Post by WhiteOwlLion »

amrrs wrote: post #57
motdiem wrote: post #39

Earlier quoted context omitted.

Seconding Webrecorder (and the newly updated WAIL) - I had the chance of meeting Ilya Kremer at a conference a few weeks ago, and I can confirm what he's doing is top notch - I'm hoping to see more work around WARC viewing and sharing in the future. (Disclaimer: I also do personal archiving stuff with getkumbu)

Is offline playback still relevant in the age of ubiquitous always connected Internet?

In my use case, some content is only available for a short window. If I want to refer to it, the URL will not work. This happens to me a lot of Wikipedia where a referenced URL is no longer working (linkrot). We need to a better way to track previous versions or access 404 pages that were previously alive.

Re: Show HN: Tesoro – Personal internet archive

#82

Post by an27 »

amrrs wrote: post #57
motdiem wrote: post #39

Earlier quoted context omitted.

Seconding Webrecorder (and the newly updated WAIL) - I had the chance of meeting Ilya Kremer at a conference a few weeks ago, and I can confirm what he's doing is top notch - I'm hoping to see more work around WARC viewing and sharing in the future. (Disclaimer: I also do personal archiving stuff with getkumbu)

Is offline playback still relevant in the age of ubiquitous always connected Internet?

Definitely, sites and content become inaccessible all the time.

For instance, I back up all new videos of my favorite YouTubers in case they are taken down (e.g. in the case of a copyright claim).

Re: Show HN: Tesoro – Personal internet archive

#83

Post by agumonkey »

flippant wrote: post #68
smoyer wrote: post #13

It's not mine unless it's running on my own servers or computer - I created a really rough version of this several years ago that is saved to my computer (and from there into box).

I wrote a similar tool which uses Electron to create PDFs of webpages and bookmark them in a SQLite database. https://github.com/marvelm/erised

Could be a browser plugin

ps: nice project btw (thanks in advance)

Re: Show HN: Tesoro – Personal internet archive

#84

Post by WhiteOwlLion »

pdfernhout wrote: post #64
rahiel wrote: post #55

Earlier quoted context omitted.

I made a browser extension [1] that automatically archives bookmarks to archive.is or (currently Chromium only) locally as MHTML files. [1]: https://github.com/rahiel/archiveror

Cool, Rahiel! Thanks for doing this. Here is a related idea I proposed a couple years ago to a Knight News Challenge on Libraries: https://web.archive.org/web/20161104175911/https://www.newsc... "Create a browser addon so when people post to the web they can send a copy for storage and hosting by a network of local libraries. ... While the Internet Archive is backing up some of the internet, it is another single poin…

What about IPFS for storing cached pages?

Re: Show HN: Tesoro – Personal internet archive

#85

Post by WhiteOwlLion »

Piskvorrr wrote: post #2

That's just as much "my own" as The Internet Archive: a website Out There somewhere. Worse, it's much more likely to rot and disappear than archive.org. Now, if I could run this locally ... (Yes, yes, `wget --convert-links`, I know. Not quite as convenient, though.)

Have you looked at WorldBrain? It is a fork of falcon, but it keeps a cache and let's you perform keyword searches against the cached content.

Re: Show HN: Tesoro – Personal internet archive

#86

Post by an27 »

rjeli wrote: post #75
j_s wrote: post #71

I would be interested in an attestation service that can provide court-admissable evidence that a particular piece of content was publically accessible on the web at a particular point in time via a particular url. I believe the only way to incentivise participation in such a system is by paying for timestamp'ed signatures, eg. "some subset of downloaded [content] from [url] at [time] hashed to [hash]" all tucked int…

Interesting - it is trivial to prove something was done today rather than yesterday, by hashing with the most recent bitcoin block or some new info. Is it possible to prove something was done in the past? All I can think of is some sort of scheme involving destroyed information.

Isn't it the other way around?

I can prove I had today's papers today, but once I've seen it I can prove it any day. So you can say "this information existed at day X or earlier".

There's also the issue of proving the content you hashed actually came from the place you say it came from. To me it seems that would require proofs of authenticity, by the source itself; not something that's easy to come by.

Re: Show HN: Tesoro – Personal internet archive

#87

Post by WhiteOwlLion »

skdotdan wrote: post #74

Nice. How are you planning to pay the servers? Your service seems quite storage-intensive.

I got a dedicated server in France that cost me less than $20 USD/month. 16GB RAM, 1TB storage: https://www.online.net/en/dedicated-server/dedibox-xc

Re: Show HN: Tesoro – Personal internet archive

#88

Post by WhiteOwlLion »

agamble wrote: post #38
Faaak wrote: post #31

Earlier quoted context omitted.

How about avoiding redundancies ? Are same CSS files cached twice or referenced by their hash ? The page URI is a bit obscure though. I think a tresoro.io/example.tld/page/foobar/timestamp would look good. What about big media content and/or small differences between them ?

Great question. Currently there is no global redundancy checking, only locally within the same page. So two CSS files which are the same from multiple archives are both kept. While this might not be ideal in terms of scaling to infinity, each archive + its dependencies are right now limited in size to 25MB, which should help keep costs under control until this is monetised. :)

What file system are you using? Couldn't a deduplicating file system handle redundancy for you?

Re: Show HN: Tesoro – Personal internet archive

#89

Post by WhiteOwlLion »

dsacco wrote: post #12
agamble wrote: post #3

Earlier quoted context omitted.

OP here. The internet archive is great, but it's not so awesome if there's some ephemeral content you need to save right away, like Tweets or social media posts. Being able to trigger an archive immediately let's you save temporary content such as that which is more prone to deletion. I'm going to build a Chrome extension to click and make cloud copy of the page you're on, hopefully that will make it seem more person…

This might sound insane, but if you modified this into a browser extension that runs locally (with options for one-off or continuous saving for entire browsing sessions) I would probably download it. Personally, I have well over 100TB of personal hard drive space in my home, and I would love to just download entire portions of my browsing history locally for archival reasons (and to truly defeat link rot). As it is n…

For one-off, you can use a bookmarklet:

For example, Wayback Machine:

Save Current URL: javascript:q=(document.location.href);void(open('http://web.archive.org/save/*/'+location.href.replace(/https..., ""),'_self ','resizable,location,menubar,toolbar,scrollbars,status'));

GoBack Current URL: javascript:q=(document.location.href);void(open('http://web.archive.org/web/*/'+location.href.replace(/https?..., ""),'_self ','resizable,location,menubar,toolbar,scrollbars,status'));

Re: Show HN: Tesoro – Personal internet archive

#90

Post by ikreymer »

JackC wrote: post #30

For personal web archiving, I highly recommend http://webrecorder.io . The site lets you download archives in standard WARC format and play them back in an offline (Electron) player. It's also open source and has a quick local setup via Docker - https://github.com/webrecorder/webrecorder . Webrecorder is by a former Internet Archive engineer, Ilya Kreymer, who now captures online performance art for an art museum. Wh…

Thanks Jack for mentioning Webrecorder! This is a project I started and it is now part of rhizome.org, a non-profit dedicated to promoting internet-based art and digital culture.

I thought I’d add a few notes here, as there’s a few ways you can use Webrecorder and related tools.

First, Webrecorder supports two distinct modes:

- Native recording mode — http/s traffic goes to through the browser and is rewritten to point to the Webrecorder server (This is the default).

- Remote browser mode — Webrecorder launches a browser in Docker container, and streams the screen to your browser (using noVNC). The traffic is either recorded or replayed depending on the mode, but the operation is the same (we call this ‘symmetrical archiving’) This gives you a recording proxy w/o having to configure your browser or install any plugins.

You can choose this mode by clicking the dropdown to choose a browser (currently Chrome and FF) This is essentially a remote browser configured via HTTP/S proxy, and allows us to record things like Flash and even Java applets, and other technologies that may become obsolete.

- We also have a desktop player app, Webrecorder Player, available for download from: https://github.com/webrecorder/webrecorderplayer-electron/re...

This is an app that plays back WARCs files (created by Webrecorder and elsewhere), and allows browsing any WARC file offline.

Another way to create a web archive (for developers): You can use the devtools in the browser to export HAR files, and Webrecorder and Webrecorder Player will convert them on the fly and play them back. Unfortunately, this option is sort of limited for developers, but you can actually create a fairly good archive locally using HAR export (available in Chrome and Firefox at least). The conversion is done using this tool: https://github.com/webrecorder/har2warc

- If you use webrecorder.io, you can register for an account or use it anonymously. If you register for an account, we provide 5GB storage and you have a permanent url for your archive. You can also upload existing WARCs (or HARs)

- You can also run Webrecorder on your own! The main codebase is at: https://github.com/webrecorder/webrecorder and the remote browser system is actually a separate component and was first used for oldweb.today and lives at https://github.com/oldweb-today

Finally, the core replay/recording tech is actually a separate component, an advanced ‘wayback machine’ being developed in https://github.com/ikreymer/pywb

There’s a lot of different components here, and we would definitely appreciate help to any and all parts of the stack if anyone is interested! All our work is open-source and we are a non-profit, so any help is appreciated.

Post reply on HN