Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Tesoro – Personal internet archive

tesoro.io

Post reply on HN
Print view View original

31–40 of 105 posts

Re: Show HN: Tesoro – Personal internet archive

#31

Post by Faaak »

agamble wrote: post #9
lozzo wrote: post #7

it would be nice to have a bit of explanation on how it works and why we can be confident that we can rely upon it

OP here. Definitely, great idea :) Briefly: Sites are archived using a system written in Golang and uploaded to a Google Cloud bucket. More: The system downloads the remote HTML, parses it to extract the relevant dependencies ( , , etc) and then downloads these as well. Tesoro is even parsing CSS files to extract the url('...') file dependencies from here as well, meaning most background images and fonts should conti…

How about avoiding redundancies ? Are same CSS files cached twice or referenced by their hash ?

The page URI is a bit obscure though. I think a tresoro.io/example.tld/page/foobar/timestamp would look good.

What about big media content and/or small differences between them ?

Re: Show HN: Tesoro – Personal internet archive

#32

Post by Sophira »

agamble wrote: post #6
detaro wrote: post #5

Earlier quoted context omitted.

You can trigger the Internet Archive manually as well.

Oh neat didn't realise that, my mistake :)

To be fair, the easiest way to do it (to my knowledge) is by direct URL entry - for example (replace 'hxxps' with 'https' for this - I didn't want crawlers to pick this up):

hxxps://web.archive.org/save/https://tesoro.io/

Re: Show HN: Tesoro – Personal internet archive

#33

Post by agamble »

prirun wrote: post #28

I think you should explain why you're paying Google to archive web pages for others, ie, how do plan on benefiting from this? If you have some business model in mind, let people know now. It's the first question that comes to my mind when someone offers a service that is free yet costs the provider real money. You obviously can't pay Google to archive everyone's web pages just for the fun of it.

OP here.

Great point. Right now this is just a single rate-limited HTML form to gauge interest. Next is to build specialty features that are worth paying for and make this sustainable. :)

Re: Show HN: Tesoro – Personal internet archive

#34

Post by agamble »

JackC wrote: post #30

For personal web archiving, I highly recommend http://webrecorder.io . The site lets you download archives in standard WARC format and play them back in an offline (Electron) player. It's also open source and has a quick local setup via Docker - https://github.com/webrecorder/webrecorder . Webrecorder is by a former Internet Archive engineer, Ilya Kreymer, who now captures online performance art for an art museum. Wh…

Thanks Jack, I hadn't heard of webrecorder before, but I'll check it out. :)

Re: Show HN: Tesoro – Personal internet archive

#35

Post by unicornporn »

Piskvorrr wrote: post #2

That's just as much "my own" as The Internet Archive: a website Out There somewhere. Worse, it's much more likely to rot and disappear than archive.org. Now, if I could run this locally ... (Yes, yes, `wget --convert-links`, I know. Not quite as convenient, though.)

https://github.com/webrecorder/webrecorder can be run using Docker. There's also plenty of Proxys that can save your browsing. See: http://netpreserve.org/projects/live-archiving-http-proxy/

Re: Show HN: Tesoro – Personal internet archive

#36

Post by crispytx »

prirun wrote: post #28

I think you should explain why you're paying Google to archive web pages for others, ie, how do plan on benefiting from this? If you have some business model in mind, let people know now. It's the first question that comes to my mind when someone offers a service that is free yet costs the provider real money. You obviously can't pay Google to archive everyone's web pages just for the fun of it.

Haters gonna hate.

Re: Show HN: Tesoro – Personal internet archive

#37

Post by shasheene »

JackC wrote: post #30

For personal web archiving, I highly recommend http://webrecorder.io . The site lets you download archives in standard WARC format and play them back in an offline (Electron) player. It's also open source and has a quick local setup via Docker - https://github.com/webrecorder/webrecorder . Webrecorder is by a former Internet Archive engineer, Ilya Kreymer, who now captures online performance art for an art museum. Wh…

Last I played with it, the latency on webrecorder was uncomfortably high for always-on recording of personal web usage (the pages only display once fully rendered). I wish webpages would render as normal and get asynchronously archived once loading is complete.

That would allow constant archival of every webpage a user ever visits -- an immutable record over the user's years of crawling the web.

Re: Show HN: Tesoro – Personal internet archive

#38

Post by agamble »

Faaak wrote: post #31
agamble wrote: post #9

Earlier quoted context omitted.

OP here. Definitely, great idea :) Briefly: Sites are archived using a system written in Golang and uploaded to a Google Cloud bucket. More: The system downloads the remote HTML, parses it to extract the relevant dependencies ( , , etc) and then downloads these as well. Tesoro is even parsing CSS files to extract the url('...') file dependencies from here as well, meaning most background images and fonts should conti…

How about avoiding redundancies ? Are same CSS files cached twice or referenced by their hash ? The page URI is a bit obscure though. I think a tresoro.io/example.tld/page/foobar/timestamp would look good. What about big media content and/or small differences between them ?

Great question.

Currently there is no global redundancy checking, only locally within the same page. So two CSS files which are the same from multiple archives are both kept. While this might not be ideal in terms of scaling to infinity, each archive + its dependencies are right now limited in size to 25MB, which should help keep costs under control until this is monetised. :)

Re: Show HN: Tesoro – Personal internet archive

#39

Post by motdiem »

JackC wrote: post #30

For personal web archiving, I highly recommend http://webrecorder.io . The site lets you download archives in standard WARC format and play them back in an offline (Electron) player. It's also open source and has a quick local setup via Docker - https://github.com/webrecorder/webrecorder . Webrecorder is by a former Internet Archive engineer, Ilya Kreymer, who now captures online performance art for an art museum. Wh…

Seconding Webrecorder (and the newly updated WAIL) - I had the chance of meeting Ilya Kremer at a conference a few weeks ago, and I can confirm what he's doing is top notch - I'm hoping to see more work around WARC viewing and sharing in the future.

(Disclaimer: I also do personal archiving stuff with getkumbu)

Re: Show HN: Tesoro – Personal internet archive

#40

Post by unicornporn »

shasheene wrote: post #37
JackC wrote: post #30

For personal web archiving, I highly recommend http://webrecorder.io . The site lets you download archives in standard WARC format and play them back in an offline (Electron) player. It's also open source and has a quick local setup via Docker - https://github.com/webrecorder/webrecorder . Webrecorder is by a former Internet Archive engineer, Ilya Kreymer, who now captures online performance art for an art museum. Wh…

Last I played with it, the latency on webrecorder was uncomfortably high for always-on recording of personal web usage (the pages only display once fully rendered). I wish webpages would render as normal and get asynchronously archived once loading is complete. That would allow constant archival of every webpage a user ever visits -- an immutable record over the user's years of crawling the web.

> That would allow constant archival of every webpage a user ever visits -- an immutable record over the user's years of crawling the web.

This is usually solved by using a proxy: http://netpreserve.org/projects/live-archiving-http-proxy/

Post reply on HN