Show HN: Pocket Stream Archive – A personal Way-Back Machine

Post by [deleted] » Fri, May 05, 2017, 7:46 PM UTC

vijucat wrote: post #48

I still use Firefox ScrapBook for this: https://addons.mozilla.org/en-US/firefox/addon/scrapbook/ Just for articles, mind you, not entire websites.

Post by zmix » Fri, May 05, 2017, 8:35 PM UTC

nikisweeting wrote: post #8

motdiem wrote: post #4

Earlier quoted context omitted.

Can one automate extensions through headless chrome ? then you might be able to trigger WarCreate instead (It will be more efficient to run the pocket export urls through WAIL though - this should give you the warcs you want)

Yeah, you can use the remote debugger protocol to make JS calls in the context of the page. https://chromedevtools.github.io/devtools-protocol/ Not sure if it's worth including in my script though, since WARCs aren't easily browseable (correct me if I'm wrong).

Post by unicornporn » Fri, May 05, 2017, 11:45 PM UTC

nikisweeting wrote: post #47

unicornporn wrote: post #42

If it would be connected to Pinboard.in as an alternative to Pocket I would be screaming with joy. :-)

If you can get me a sample pinboard export to look at, I'll whip up a regex that makes it work.

Post by arkenflame » Sat, May 06, 2017, 1:08 AM UTC

Post by ianbicking » Sat, May 06, 2017, 6:04 PM UTC

nikisweeting wrote: post #39

driverdan wrote: post #34

Earlier quoted context omitted.

PDFs are really not suitable for archiving websites since they're designed around pages and the web does not have pages. A better option is to render a page with JS turned on and save the resulting HTML.

I was doing this before with the chrome --dump-dom flag but the output I was getting was garbage, and no more useful than the simpler wget download. PDF turned out to produce really nice, readable archives about 75% of the time, so I kept it in. Text-based sites tend to do a good job of having PDF-friendly styling.