Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Pocket Stream Archive – A personal Way-Back Machine

github.com

Post reply on HN
Print view View original

31–40 of 70 posts

Re: Show HN: Pocket Stream Archive – A personal Way-Back Machine

#31

Post by nikisweeting »

jccalhoun wrote: post #28
jl6 wrote: post #19

Screenshotting or PDFing of a website is an increasingly important archiving tool, to supplement wget. I've come across a lot of websites that won't render any content if not connected to a live server.

Agreed. I research new media and archive.org is invaluable to me. I worry that current web sites won't be able to be preserved. (much like many of the flash sites and real audio of the past are largely gone.)

Speaking of audio, one of my highest priority TODO's for this project is to use youtube-dl for nicely archiving youtube & soundcloud links.

Re: Show HN: Pocket Stream Archive – A personal Way-Back Machine

#32

Post by MrRadar »

nikisweeting wrote: post #8
motdiem wrote: post #4

Earlier quoted context omitted.

Can one automate extensions through headless chrome ? then you might be able to trigger WarCreate instead (It will be more efficient to run the pocket export urls through WAIL though - this should give you the warcs you want)

Yeah, you can use the remote debugger protocol to make JS calls in the context of the page. https://chromedevtools.github.io/devtools-protocol/ Not sure if it's worth including in my script though, since WARCs aren't easily browseable (correct me if I'm wrong).

> Not sure if it's worth including in my script though, since WARCs aren't easily browseable (correct me if I'm wrong).

I've had good luck with WebArchivePlayer: https://github.com/ikreymer/webarchiveplayer

Re: Show HN: Pocket Stream Archive – A personal Way-Back Machine

#33

Post by anc84 »

bicubic wrote: post #25
nikisweeting wrote: post #22

Earlier quoted context omitted.

Slowing down the inevitable tide of https://en.wikipedia.org/wiki/Link_rot . When I cite blog posts or want to share sites that have gone down, I can swap out the links for my archived versions.

Why is archive.org or one of the other centralized web archives not suitable for that? They don't index the content you want to retain?

Never rely on others to care about things you need.

Re: Show HN: Pocket Stream Archive – A personal Way-Back Machine

#34

Post by driverdan »

jl6 wrote: post #19

Screenshotting or PDFing of a website is an increasingly important archiving tool, to supplement wget. I've come across a lot of websites that won't render any content if not connected to a live server.

PDFs are really not suitable for archiving websites since they're designed around pages and the web does not have pages.

A better option is to render a page with JS turned on and save the resulting HTML.

Re: Show HN: Pocket Stream Archive – A personal Way-Back Machine

#35

Post by TazeTSchnitzel »

driverdan wrote: post #34
jl6 wrote: post #19

Screenshotting or PDFing of a website is an increasingly important archiving tool, to supplement wget. I've come across a lot of websites that won't render any content if not connected to a live server.

PDFs are really not suitable for archiving websites since they're designed around pages and the web does not have pages. A better option is to render a page with JS turned on and save the resulting HTML.

PDFs have the advantage of being a fixed format that should display the same everywhere. It's probably less fragile than modern CSS or HTML.

However, as you point out, PDFs are designed around the printed page, not the flowing arbitrary-page-size documents of the web.

Re: Show HN: Pocket Stream Archive – A personal Way-Back Machine

#36

Post by no_news_is »

nikisweeting wrote: post #9
ents wrote: post #7

Would be cool to see this for Instapaper or Pinboard

My script should work with very minimal tweaking if you can get a list of urls + titles from those services. Just one line of regex changes probably: https://github.com/pirate/pocket-archive-stream/blob/master/...

Just about what I was hoping to see in the comments, was the expected format of the input.

May I suggest including in the readme.md, a sample line of the Pocket export format/your input format?

Thanks for publishing!

Re: Show HN: Pocket Stream Archive – A personal Way-Back Machine

#37

Post by nikisweeting »

no_news_is wrote: post #36
nikisweeting wrote: post #9

Earlier quoted context omitted.

My script should work with very minimal tweaking if you can get a list of urls + titles from those services. Just one line of regex changes probably: https://github.com/pirate/pocket-archive-stream/blob/master/...

Just about what I was hoping to see in the comments, was the expected format of the input. May I suggest including in the readme.md, a sample line of the Pocket export format/your input format? Thanks for publishing!

I included a sample of the expected pocket list format in the repo: https://github.com/pirate/pocket-archive-stream/blob/master/...

And a comment next to the regex for parsing it: https://github.com/pirate/pocket-archive-stream/blob/master/...

Re: Show HN: Pocket Stream Archive – A personal Way-Back Machine

#39

Post by nikisweeting »

driverdan wrote: post #34
jl6 wrote: post #19

Screenshotting or PDFing of a website is an increasingly important archiving tool, to supplement wget. I've come across a lot of websites that won't render any content if not connected to a live server.

PDFs are really not suitable for archiving websites since they're designed around pages and the web does not have pages. A better option is to render a page with JS turned on and save the resulting HTML.

I was doing this before with the chrome --dump-dom flag but the output I was getting was garbage, and no more useful than the simpler wget download. PDF turned out to produce really nice, readable archives about 75% of the time, so I kept it in. Text-based sites tend to do a good job of having PDF-friendly styling.

Re: Show HN: Pocket Stream Archive – A personal Way-Back Machine

#40

Post by nikisweeting »

fiatjaf wrote: post #38

You see something is flawed in Redux at the point you have to pass strings (uppercase constants defined somewhere) around, import them in every file, pass them as identifiers of what you should do with each data. Strings!

Did you comment on the wrong article by accident? https://news.ycombinator.com/item?id=14273549
Post reply on HN