Ask HN: CLI tool to download webpages and convert to Markdown?

#1

Post by raytracer » Fri, Jun 12, 2020, 3:29 AM UTC

Is there a CLI tool that downloads webpages and converts them to Markdown?

I've just started using Obsidian https://obsidian.md/ and would like a way to save interesting blog posts and articles.

Re: Ask HN: CLI tool to download webpages and convert to Markdown?

#2

Post by miles » Fri, Jun 12, 2020, 3:42 AM UTC

  curl --silent https://example.com/foo.html | pandoc --from html --to markdown_strict -o foo.md

From Converting HTML to Markdown using Pandoc http://www.cantoni.org/2019/01/27/converting-html-markdown-u...

Re: Ask HN: CLI tool to download webpages and convert to Markdown?

#3

Post by longimanus » Fri, Jun 12, 2020, 7:34 AM UTC

I find Joplin https://joplinapp.org does a good job of producing markdown from web pages and already has sync capability built in. Looks like it would be ideal for working with Obsidian

Re: Ask HN: CLI tool to download webpages and convert to Markdown?

#4

Post by longimanus » Fri, Jun 12, 2020, 7:53 AM UTC

longimanus wrote: post #3

I find Joplin https://joplinapp.org does a good job of producing markdown from web pages and already has sync capability built in. Looks like it would be ideal for working with Obsidian

Exported my Joplin markdown and opened it up in Obsidian. Works like a dream and the best part is Jopli already has it's own web clipper. Seems like a superb match

Re: Ask HN: CLI tool to download webpages and convert to Markdown?

#5

Post by I_complete_me » Fri, Jun 12, 2020, 8:24 AM UTC

miles wrote: post #2

curl --silent https://example.com/foo.html | pandoc --from html --to markdown_strict -o foo.md From Converting HTML to Markdown using Pandoc http://www.cantoni.org/2019/01/27/converting-html-markdown-u...

I tried that on a HackerNews page and the content of the output file foo.md was in html.

Re: Ask HN: CLI tool to download webpages and convert to Markdown?

#6

Post by miles » Fri, Jun 12, 2020, 8:49 AM UTC

I_complete_me wrote: post #5

miles wrote: post #2

curl --silent https://example.com/foo.html | pandoc --from html --to markdown_strict -o foo.md From Converting HTML to Markdown using Pandoc http://www.cantoni.org/2019/01/27/converting-html-markdown-u...

I tried that on a HackerNews page and the content of the output file foo.md was in html.

Sorry, had only tested it on my own site, which worked as expected:

  curl --silent https://tinyapps.org/ | pandoc --from html --to markdown_strict -o index.md

Re: Ask HN: CLI tool to download webpages and convert to Markdown?

#7

Post by anandology » Fri, Jun 12, 2020, 9:57 AM UTC

https://github.com/aaronsw/html2text

Re: Ask HN: CLI tool to download webpages and convert to Markdown?

#8

Post by crazypython » Fri, Jun 12, 2020, 12:22 PM UTC

miles wrote: post #2

curl --silent https://example.com/foo.html | pandoc --from html --to markdown_strict -o foo.md From Converting HTML to Markdown using Pandoc http://www.cantoni.org/2019/01/27/converting-html-markdown-u...

I'd like to add that Calibre (ebook-convert on the CLI) has a mode for outputting a .txt file with markdown formatting. It can take HTML as input.

Re: Ask HN: CLI tool to download webpages and convert to Markdown?

#9

Post by dougskinner » Fri, Jun 12, 2020, 1:16 PM UTC

I made this a couple weeks ago: https://www.npmjs.com/package/@dougskinner/markdowner

Re: Ask HN: CLI tool to download webpages and convert to Markdown?

#10

Post by miles » Fri, Jun 12, 2020, 2:12 PM UTC

anandology wrote: post #7

https://github.com/aaronsw/html2text

Thank you, Anand. Despite being 8 years old, Aaron's html2text.py worked perfectly to convert the HN homepage to Markdown. His memory (and code) continues to be a blessing!