Could you add an option to either add tagging, or separate the tagged items into folders?
e.g. "programming/", "docker/" etc, I often find myself digging through my Pocket archive trying to find that one article I found 6 months ago and it gets incredibly annoying
This is cool. Could you add an option to either add tagging, or separate the tagged items into folders? e.g. "programming/", "docker/" etc, I often find myself digging through my Pocket archive trying to find that one article I found 6 months ago and it gets incredibly annoying
I like having the sites by timestamp because they're guaranteed to be unique, and it makes traversing them easy. I'd be happy to add a tag column to the index though, which you could use with Ctrl+F to find articles. https://github.com/pirate/pocket-archive-stream/issues/1
Now if only Chromium could learn to write WARC archives, then it would be on par! :) Great project!
I've been thinking along those very same lines for a long time (this project makes me wish for more free time).
I have half a mind to fork this and add something like https://github.com/internetarchive/warcprox, or at the very least walk through the generated HTML and brute-force inline all assets as a first pass :)
Can one automate extensions through headless chrome ? then you might be able to trigger WarCreate instead (It will be more efficient to run the pocket export urls through WAIL though - this should give you the warcs you want)
Yeah, you can use the remote debugger protocol to make JS calls in the context of the page. https://chromedevtools.github.io/devtools-protocol/ Not sure if it's worth including in my script though, since WARCs aren't easily browseable (correct me if I'm wrong).
Well, no, but on a Mac .webarchives are Spotlight indexable and make for a nice single-file archiving approach, and I might actually have some old code that tries to convert between the two...
Can one automate extensions through headless chrome ? then you might be able to trigger WarCreate instead (It will be more efficient to run the pocket export urls through WAIL though - this should give you the warcs you want)
Yeah, you can use the remote debugger protocol to make JS calls in the context of the page. https://chromedevtools.github.io/devtools-protocol/ Not sure if it's worth including in my script though, since WARCs aren't easily browseable (correct me if I'm wrong).
Yeah, you can use the remote debugger protocol to make JS calls in the context of the page. https://chromedevtools.github.io/devtools-protocol/ Not sure if it's worth including in my script though, since WARCs aren't easily browseable (correct me if I'm wrong).
Well, no, but on a Mac .webarchives are Spotlight indexable and make for a nice single-file archiving approach, and I might actually have some old code that tries to convert between the two...
Screenshotting or PDFing of a website is an increasingly important archiving tool, to supplement wget. I've come across a lot of websites that won't render any content if not connected to a live server.
Well, no, but on a Mac .webarchives are Spotlight indexable and make for a nice single-file archiving approach, and I might actually have some old code that tries to convert between the two...
Oh sweet, that's pretty nice.
Also, I've been hacking away at http://github.com/rcarmo/newsfeed-corpus for a bit. Hadn't thought of doing archival on everything, but having this tied in seems like a logical step ;)