Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Tesoro – Personal internet archive

tesoro.io

Post reply on HN
Print view View original

21–30 of 105 posts

Re: Show HN: Tesoro – Personal internet archive

#21

Post by 19eightyfour »

agamble wrote: post #9
lozzo wrote: post #7

it would be nice to have a bit of explanation on how it works and why we can be confident that we can rely upon it

OP here. Definitely, great idea :) Briefly: Sites are archived using a system written in Golang and uploaded to a Google Cloud bucket. More: The system downloads the remote HTML, parses it to extract the relevant dependencies ( , , etc) and then downloads these as well. Tesoro is even parsing CSS files to extract the url('...') file dependencies from here as well, meaning most background images and fonts should conti…

The issue is cost. Your costs are disk space for people's archives, instances for people's use, and bandwidth for the fetches and crawls and access.

How can you pay for this if it's free? It's unreliable unless its financially viable.

Re: Show HN: Tesoro – Personal internet archive

#22

Post by agamble »

venning wrote: post #19

Thoughts: I like the look. Very clean. I like how fast it's responding; better than archive.org (though, obviously, they have different scaling problems). "Your own internet archive" might be overselling it, as other commenters have pointed out; the "Your" feels a bit misleading. I think "Save a copy of any webpage." gives a better impression, which you use on the site itself. The "Archive!" link probably shouldn't w…

Thanks! These are great comments - I'll look into the issue with saving Hacker News CSS + JS.

Re: Show HN: Tesoro – Personal internet archive

#23

Post by agamble »

19eightyfour wrote: post #21
agamble wrote: post #9

Earlier quoted context omitted.

OP here. Definitely, great idea :) Briefly: Sites are archived using a system written in Golang and uploaded to a Google Cloud bucket. More: The system downloads the remote HTML, parses it to extract the relevant dependencies ( , , etc) and then downloads these as well. Tesoro is even parsing CSS files to extract the url('...') file dependencies from here as well, meaning most background images and fonts should conti…

The issue is cost. Your costs are disk space for people's archives, instances for people's use, and bandwidth for the fetches and crawls and access. How can you pay for this if it's free? It's unreliable unless its financially viable.

Totally right, great observation :)

For now it's a free service with a single rate-limited form. Now it's time to work on adding specialty features that are worth paying for.

Re: Show HN: Tesoro – Personal internet archive

#24

Post by agamble »

unicornporn wrote: post #20

In what way could this considered to be “your own internet archive”? I see no way to register a user and save pages to a collection. If you really want to create your own archive, set up a Live Archiving HTTP Proxy[1], run SquidMan [2] or check out WWWOFFLE[3]. If you want something simpler, have a look at Webrecorder[4] or a paid Pinboard account with the “Bookmark Archive”[5]. [1] http://netpreserve.org/projects/li…

Great points.

You're right, for now it's a single rate-limited HTML form and you'll have to manually collate the links to the archives you create. I'll be adding specialty features (with accounts) next. :)

Re: Show HN: Tesoro – Personal internet archive

#26

Post by agamble »

dbz wrote: post #15

This is pretty cool. I have a chrome extension that let's you view the cached version of a web page [1]. Would I be able to use this through an API? I currently support Google Cache, WayBack Machine, and CoralCDN, but Coral doesn't work well and I'd like to replace it with something else. [1] https://chrome.google.com/webstore/detail/cmmlgikpahieigpccl...

OP here.

Yup, API and chrome extension are next on the feature list. :)

Re: Show HN: Tesoro – Personal internet archive

#27

Post by jpalomaki »

This might be a good use case for distributed storage (IPFS?).

Instead of hosting this directly on my computer, it would be interesting to have a setup where the archiving is done via the service and I would just provide somewhere a storage space where the content would end up being mirrored (just to guarantee that my valuable things are saved at least somewhere, should the the other nodes decide to remove the content).

I would prefer this setup, because it would be easily accessible for me from any device and I would not need to worry about running some always available system. With some suitable P2P setup my storage node would have less strict uptime requirements.

Re: Show HN: Tesoro – Personal internet archive

#28

Post by prirun »

I think you should explain why you're paying Google to archive web pages for others, ie, how do plan on benefiting from this? If you have some business model in mind, let people know now. It's the first question that comes to my mind when someone offers a service that is free yet costs the provider real money. You obviously can't pay Google to archive everyone's web pages just for the fun of it.

Re: Show HN: Tesoro – Personal internet archive

#29

Post by rahiel »

An internet archive can only provide value if it's there for the long-term. What's your plan to keep this service running if it gets popular? For example, archive.is costed about $2000/month at the start of 2014 [1]. I expect it to cost even more now.

[1]: http://blog.archive.is/post/72136308644/how-much-does-it-cos...

Re: Show HN: Tesoro – Personal internet archive

#30

Post by JackC »

For personal web archiving, I highly recommend http://webrecorder.io. The site lets you download archives in standard WARC format and play them back in an offline (Electron) player. It's also open source and has a quick local setup via Docker - https://github.com/webrecorder/webrecorder .

Webrecorder is by a former Internet Archive engineer, Ilya Kreymer, who now captures online performance art for an art museum. What he's doing with capture and playback of Javascript, web video, streaming content, etc. is state of the art as far as I know.

(Disclaimer - I use bits of Webrecorder for my own archive, perma.cc.)

For OP, I would say consider building on and contributing back to Webrecorder -- or alternatively figure out what Webrecorder is good at and make sure you're good at something different. It's a crazy hard problem to do well and it's great to have more ideas in the mix.

Post reply on HN