Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Tesoro – Personal internet archive

tesoro.io

Post reply on HN
Print view View original

71–80 of 105 posts

Re: Show HN: Tesoro – Personal internet archive

#71

Post by j_s »

I would be interested in an attestation service that can provide court-admissable evidence that a particular piece of content was publically accessible on the web at a particular point in time via a particular url.

I believe the only way to incentivise participation in such a system is by paying for timestamp'ed signatures, eg. "some subset of downloaded [content] from [url] at [time] hashed to [hash]" all tucked into a Bitcoin transaction or something. There are services that will do this with user-provided content[1]; I am looking for something that will pull a url and timestamp the content.

This would also be a way to detect when different users are being served different content at the same url, thus the need for a global network of validators.

[1] https://proofofexistence.com/

Re: Show HN: Tesoro – Personal internet archive

#72

Post by owenversteeg »

JackC wrote: post #30

For personal web archiving, I highly recommend http://webrecorder.io . The site lets you download archives in standard WARC format and play them back in an offline (Electron) player. It's also open source and has a quick local setup via Docker - https://github.com/webrecorder/webrecorder . Webrecorder is by a former Internet Archive engineer, Ilya Kreymer, who now captures online performance art for an art museum. Wh…

Wow, webrecorder seems very cool, especially since it's OSS. Is there any way to set it up to record all incoming traffic? In these days of cheap storage that'd be very cool. I know I personally only use about 30GB a month, so a $70 2TB hard drive would last me five and a half years of browsing.

Re: Show HN: Tesoro – Personal internet archive

#73

Post by owenversteeg »

unicornporn wrote: post #40
shasheene wrote: post #37

Earlier quoted context omitted.

Last I played with it, the latency on webrecorder was uncomfortably high for always-on recording of personal web usage (the pages only display once fully rendered). I wish webpages would render as normal and get asynchronously archived once loading is complete. That would allow constant archival of every webpage a user ever visits -- an immutable record over the user's years of crawling the web.

> That would allow constant archival of every webpage a user ever visits -- an immutable record over the user's years of crawling the web. This is usually solved by using a proxy: http://netpreserve.org/projects/live-archiving-http-proxy/

Can this be combined with webrecorder? Does anyone know someone who's done this? I only use about 30GB of traffic a month so a 2TB $70 hard drive would last me almost six years.

Re: Show HN: Tesoro – Personal internet archive

#75

Post by rjeli »

j_s wrote: post #71

I would be interested in an attestation service that can provide court-admissable evidence that a particular piece of content was publically accessible on the web at a particular point in time via a particular url. I believe the only way to incentivise participation in such a system is by paying for timestamp'ed signatures, eg. "some subset of downloaded [content] from [url] at [time] hashed to [hash]" all tucked int…

Interesting - it is trivial to prove something was done today rather than yesterday, by hashing with the most recent bitcoin block or some new info.

Is it possible to prove something was done in the past? All I can think of is some sort of scheme involving destroyed information.

Re: Show HN: Tesoro – Personal internet archive

#77

Post by dddw »

pbhjpbhj wrote: post #17
smoyer wrote: post #13

It's not mine unless it's running on my own servers or computer - I created a really rough version of this several years ago that is saved to my computer (and from there into box).

I adapted a bash script someone posted here, it uses Firefox bookmarks (pages.sqlite). Cron runs the script and downloads every page I've bookmarked that month (after some filtering). I don't use it often but sometimes I'll awk-grep it; I'm a hoarder in real life too!

indeed interested in this too

Re: Show HN: Tesoro – Personal internet archive

#78

Post by pbhjpbhj »

iso-8859-1 wrote: post #43
pbhjpbhj wrote: post #25

How are you handling copyright infringement? Outside USAs Fair Use terms this looks like pretty blatant infringement.

What is there to handle? You take down stuff when you get an email? Most users of this will be so small that they'll never get noticed. Maybe they won't even be online, how are you going to know you were infringed? Maybe the crawler allows for spoofing the user-agent.

So, ignoring it basically.

If a person in the UK uses your service you're committing contributory infringement for commercial purposes, AFAICT.

Moreover, the ECD has different protections than DMCA. Particularly getting a takedown notice isn't required.

>Maybe the crawler allows for spoofing the user-agent. //

As a tort you only need to get a preponderance of evidence. IP of the crawler that made the copy puts the owner of that IP in court for contributory infringement, no?

If you make copies of parts of BBC sites and serve those copies from your server how is that not copyright infringement by you??

FWIW I like the service and do not like the copyright regime as it stands, particularly how UK law lacks the breadth of liberties of Fair Use.

Re: Show HN: Tesoro – Personal internet archive

#79

Post by [deleted] »

rjeli wrote: post #75
j_s wrote: post #71

I would be interested in an attestation service that can provide court-admissable evidence that a particular piece of content was publically accessible on the web at a particular point in time via a particular url. I believe the only way to incentivise participation in such a system is by paying for timestamp'ed signatures, eg. "some subset of downloaded [content] from [url] at [time] hashed to [hash]" all tucked int…

Interesting - it is trivial to prove something was done today rather than yesterday, by hashing with the most recent bitcoin block or some new info. Is it possible to prove something was done in the past? All I can think of is some sort of scheme involving destroyed information.

[deleted]

Re: Show HN: Tesoro – Personal internet archive

#80

Post by JackC »

owenversteeg wrote: post #72
JackC wrote: post #30

For personal web archiving, I highly recommend http://webrecorder.io . The site lets you download archives in standard WARC format and play them back in an offline (Electron) player. It's also open source and has a quick local setup via Docker - https://github.com/webrecorder/webrecorder . Webrecorder is by a former Internet Archive engineer, Ilya Kreymer, who now captures online performance art for an art museum. Wh…

Wow, webrecorder seems very cool, especially since it's OSS. Is there any way to set it up to record all incoming traffic? In these days of cheap storage that'd be very cool. I know I personally only use about 30GB a month, so a $70 2TB hard drive would last me five and a half years of browsing.

If you're a coder I bet you could hack it to do that. It has an amazing containerized browser mode where you can browse in a remote browser via VNC, with the remote browser set up to use a WARC-writing proxy. So the general outline would be to run it locally in Docker; expose the proxy port used by the containerized browsers; and configure your own browsers to use the same proxy.

I'm not sure how much this would interfere with normal browsing -- it's not a typical usecase.

Post reply on HN