Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Pocket Stream Archive – A personal Way-Back Machine

github.com

Post reply on HN
Print view View original

21–30 of 70 posts

Re: Show HN: Pocket Stream Archive – A personal Way-Back Machine

#21

Post by throwaway7767 »

rcarmo wrote: post #14
anc84 wrote: post #3

Now if only Chromium could learn to write WARC archives, then it would be on par! :) Great project!

I've been thinking along those very same lines for a long time (this project makes me wish for more free time). I have half a mind to fork this and add something like https://github.com/internetarchive/warcprox , or at the very least walk through the generated HTML and brute-force inline all assets as a first pass :)

I've been thinking I'd love to have a WARC archive of all my browsing. So many times sites I remember seeing have gone offline, and didn't get archived by the big services. Ideally this has to happen with browser cooperation, so it can save resources from complex dynamic pages, including responses to user action.

This must happen either in the browser or in a proxy like the linked warcprox, in order to catch everything. But the proxy solution is getting less practical every day with key pinning and HSTS.

Maybe a future firefox will have an option to export everything to WARC?

Re: Show HN: Pocket Stream Archive – A personal Way-Back Machine

#22

Post by nikisweeting »

bicubic wrote: post #18

This seems neat, curious what are the use cases for this?

Slowing down the inevitable tide of https://en.wikipedia.org/wiki/Link_rot. When I cite blog posts or want to share sites that have gone down, I can swap out the links for my archived versions.

Re: Show HN: Pocket Stream Archive – A personal Way-Back Machine

#23

Post by nikisweeting »

jl6 wrote: post #19

Screenshotting or PDFing of a website is an increasingly important archiving tool, to supplement wget. I've come across a lot of websites that won't render any content if not connected to a live server.

I couldn't agree more. I wish more sites would load without needing multiple seconds of JS execution and AJAX. One of my TODOs is to get full-page screenshots working as well.

Re: Show HN: Pocket Stream Archive – A personal Way-Back Machine

#24

Post by nikisweeting »

rcarmo wrote: post #14
anc84 wrote: post #3

Now if only Chromium could learn to write WARC archives, then it would be on par! :) Great project!

I've been thinking along those very same lines for a long time (this project makes me wish for more free time). I have half a mind to fork this and add something like https://github.com/internetarchive/warcprox , or at the very least walk through the generated HTML and brute-force inline all assets as a first pass :)

I would be very on-board with adding a warc exporting option. I also hate how Chrome tosses all history older than 3 months. Running an archiving proxy hooked up to archive.py would kill both birds with one stone.

Re: Show HN: Pocket Stream Archive – A personal Way-Back Machine

#25

Post by bicubic »

nikisweeting wrote: post #22
bicubic wrote: post #18

This seems neat, curious what are the use cases for this?

Slowing down the inevitable tide of https://en.wikipedia.org/wiki/Link_rot . When I cite blog posts or want to share sites that have gone down, I can swap out the links for my archived versions.

Why is archive.org or one of the other centralized web archives not suitable for that? They don't index the content you want to retain?

Re: Show HN: Pocket Stream Archive – A personal Way-Back Machine

#26

Post by AznHisoka »

nikisweeting wrote: post #23
jl6 wrote: post #19

Screenshotting or PDFing of a website is an increasingly important archiving tool, to supplement wget. I've come across a lot of websites that won't render any content if not connected to a live server.

I couldn't agree more. I wish more sites would load without needing multiple seconds of JS execution and AJAX. One of my TODOs is to get full-page screenshots working as well.

in some cases it leads to worse horrible experiences like my Linkedin feed. I have to wait a few seconss everytime before content loads.

Re: Show HN: Pocket Stream Archive – A personal Way-Back Machine

#27

Post by nikisweeting »

bicubic wrote: post #25
nikisweeting wrote: post #22

Earlier quoted context omitted.

Slowing down the inevitable tide of https://en.wikipedia.org/wiki/Link_rot . When I cite blog posts or want to share sites that have gone down, I can swap out the links for my archived versions.

Why is archive.org or one of the other centralized web archives not suitable for that? They don't index the content you want to retain?

Archive.org is a single-point-of-failure, also they don't take PDFs and screenshots of fully-rendered sites with their JS-loaded content.

Re: Show HN: Pocket Stream Archive – A personal Way-Back Machine

#28

Post by jccalhoun »

jl6 wrote: post #19

Screenshotting or PDFing of a website is an increasingly important archiving tool, to supplement wget. I've come across a lot of websites that won't render any content if not connected to a live server.

Agreed. I research new media and archive.org is invaluable to me. I worry that current web sites won't be able to be preserved. (much like many of the flash sites and real audio of the past are largely gone.)

Re: Show HN: Pocket Stream Archive – A personal Way-Back Machine

#30

Post by nikisweeting »

antman wrote: post #29

The demo does not have images. Maybe try wget -nc -np -E -H -k -K -p -U 'Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1.6) Gecko/20070802 SeaMonkey/1.1.4' -e robots=off

I opted not to download images using wget. I figured if I needed in-article images the PDF+screenshot would be enough.
Post reply on HN