Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: I made a tool to clean and convert any webpage to Markdown

markdowndown.vercel.app

Post reply on HN
Print view View original

41–50 of 107 posts

Re: Show HN: I made a tool to clean and convert any webpage to Markdown

#41

Post by thangalin »

rubyn00bie wrote: post #33
blobcode wrote: post #30

This is one of those things that the ever-amazing pandoc ( https://pandoc.org/ ) does very well, on top of supporting virtually every other document format.

I second this. Pandoc is up there as one of the most useful tools that exist, that almost no one talks about. It's amazing, easy to use, and works. I regularly see new tools in the space pop-up, but someone would have to have a REALLY unique and compelling feature, or highly optimized use case to get me to use anything else (besides Pandoc).

I wrote a series of blog posts about typesetting Markdown using pandoc:

https://dave.autonoma.ca/blog

Eventually, I found pandoc to be a little limiting:

* Awkward to use interpolated variables within prose.

* No real-time preview prior to rendering the final document.

* Limited options for TeX support (e.g., SVG vs. inline; ConTeXt vs. LaTeX).

* Inconsistent syntax for captions and cross-references.

* Requires glue to apply a single YAML metadata source file to multiple documents (e.g., book chapters).

* Does not (reliably) convert straight quotes to curly quotes.

For my purposes, I wanted to convert variable-laden Markdown and R Markdown to text, XHTML, and PDF formats. Eventually I replaced my tool chain of yamlp + pandoc + knitr by writing an integrated FOSS cross-platform desktop editor.

https://keenwrite.com/

KeenWrite uses flexmark-java + Renjin + KeenTeX + KeenQuotes to provide a solution that can replace pandoc + knitr in some situations.

Note how the captions and cross-reference syntax for images, tables, and equations is unified to use a double-colon sigil:

https://gitlab.com/DaveJarvis/KeenWrite/-/blob/main/docs/ref...

There's also command-line usage for integrating into build pipelines:

https://gitlab.com/DaveJarvis/KeenWrite/-/blob/main/docs/cmd...

Re: Show HN: I made a tool to clean and convert any webpage to Markdown

#42

Post by Terretta »

smartmic wrote: post #11
ChadNauseam wrote: post #5

This is awesome. I kind of want a browser extension that does this to every page I read and saves them somewhere.

My choice (manual): Markdown clipper https://github.com/deathau/markdown-clipper I guess there are dozens of alternative extensions available out there …

This fork:

https://github.com/deathau/markdownload

With extension available for Firefox, Google Chrome, Microsoft Edge and Safari.

Re: Show HN: I made a tool to clean and convert any webpage to Markdown

#43

Post by hombre_fatal »

chrisweekly wrote: post #20
IanCal wrote: post #15

Earlier quoted context omitted.

How do you achieve the same things without AI here using that tool?

"How do you do it without AI" is a question I (sadly) expect to see more often.

Meh, it’s just the “how does it work?” question. How content extractors work is interesting and not obvious nor trivial.

And even when you see how readability parser works, AI handles most of the edge cases that content extractors fail on, so they are genuinely superseded by LLMs.

Re: Show HN: I made a tool to clean and convert any webpage to Markdown

#44

Post by fbdab103 »

LeonidBugaev wrote: post #3

One of the cases when AI not needed. There is very good working algorithm to extract content from the pages, one of implementations: https://github.com/buriy/python-readability

I was honestly expecting it to be mostly black magic, but it looks like the meat of the project is a bunch of (surely hard won) regexes. Nifty.

Re: Show HN: I made a tool to clean and convert any webpage to Markdown

#45

Post by fbdab103 »

cratermoon wrote: post #6

I've found htmltidy [1] and pandoc html->markdown sufficiently capable. 1 http://www.html-tidy.org/ 2 https://pandoc.org/

Never heard of tidy, this looks promising.

I am kind of tempted/horrified to run all of my final templated HTML through this and see if I can spot any lingering malformations. Depending on how structured the corrections are, could make it a test-suite thing.

Re: Show HN: I made a tool to clean and convert any webpage to Markdown

#46

Post by jot »

Great idea to offer image downloads and filtering with GPT!

I built a similar tool last year that doesn't have those features: https://url2text.com/

Apologies if the UI is slow - you can see some example output on the homepage.

The API it's built on is Urlbox's website screenshot API which performs far better when used directly. You can request markdown along with JS rendered HTML, metadata and screenshot all in one go: https://urlbox.com/extracting-text

You can even have it all saved directly to your S3-compatible storage: https://urlbox.com/s3

And/or delivered by webhook: https://urlbox.com/webhooks

I've been running over 1 million renders per month using Urlbox's markdown feature for a side project. It's so much better using markdown like this for embeddings and in prompts.

If you want to scrape whole websites like this you might also want to checkout this new tool by dctanner: https://usescraper.com/

Re: Show HN: I made a tool to clean and convert any webpage to Markdown

#48

Post by old_dev »

It looks like, if the website presents a cookie message, the tool just gets stuck on that and does not parse the actual content. As an example, I tried https://www.cnbc.com/ and all it created was a markdown of the cookie message and some legalese around it.

Re: Show HN: I made a tool to clean and convert any webpage to Markdown

#50

Post by jot »

old_dev wrote: post #48

It looks like, if the website presents a cookie message, the tool just gets stuck on that and does not parse the actual content. As an example, I tried https://www.cnbc.com/ and all it created was a markdown of the cookie message and some legalese around it.

It's not easy working around things like that. But here's how it could work: https://url2text.com/u/wYVake

We were lucky to build this on a mature API that already solves loads of the edge cases around rendering different kinds of pages.

Post reply on HN