A year ago I implemented this as well (albeit as a commercial offering with 100 free scrapes per month): https://2markdown.com
It also has javascript-enabled browsing available in private beta. Will make it public this week. In my experience, people fall back to the simple scraping and not use js that much, if at all.
Show HN: I made a tool to clean and convert any webpage to Markdown
81–90 of 107 posts
Re: Show HN: I made a tool to clean and convert any webpage to Markdown
#82Just tried it on a complex marketing page and it did a great job. Congrats! I'm curious, if you care to share, what kind of load this places on your host? Is this something that you can keep going for free or will it eventually become non-cost efficient to keep running?
this is slightly heavy due to loading a headless chrome instance. I will look into optimizing this part.
other than that, gpt4 is expensive, but so far it’s been negligible so i am hopeful.
i feel i can keep it around for long.
Re: Show HN: I made a tool to clean and convert any webpage to Markdown
#83nitpick: the tooltip (triggered by the question mark icon) does not work on mobile - at least on my iPhone (both chrome & safari)
May be worth taking a look at.
Good stuff otherwise! Cheers on the launch
Re: Show HN: I made a tool to clean and convert any webpage to Markdown
#84Earlier quoted context omitted.
it uses readibility but does some additional stuff like relink images to local paths etc., which I needed
I have had challenges with readability. The output is good for blogs but when we try it for other type of content, it misses on important details even when the page is quite text-heavy just like blog.
yeah that’s correct. i put a checkbox to disable readability filter if needed…
Re: Show HN: I made a tool to clean and convert any webpage to Markdown
#85Converting websites to markdown comes with 3 distinct problems: 1. Throughly scraping the content of page (high recall) 2. Dropping all the ads/auxilliary content (high precision) 3. And getting the correct layout/section types (formatting) For #2 and #3 - Trafilatura, Newspaper4k and python-readability based solutions work best out of the box. For #1, any scraping service + selenium is going to do a great job. Could…
Thoroughly scraping is challenging, especially in an environment where you don’t have (or want) a JavaScript runtime.
For content extraction, I found the approach the Postlight library takes quite neat. It scores individual html nodes based on some heuristics (text length, link density, css classes). It the selects the nodes with the highest score. [1] I ported it to Swift for a personal read later app.
Re: Show HN: I made a tool to clean and convert any webpage to Markdown
#86Converting websites to markdown comes with 3 distinct problems: 1. Throughly scraping the content of page (high recall) 2. Dropping all the ads/auxilliary content (high precision) 3. And getting the correct layout/section types (formatting) For #2 and #3 - Trafilatura, Newspaper4k and python-readability based solutions work best out of the box. For #1, any scraping service + selenium is going to do a great job. Could…
Thoroughly scraping is challenging, especially in an environment where you don’t have (or want) a JavaScript runtime. For content extraction, I found the approach the Postlight library takes quite neat. It scores individual html nodes based on some heuristics (text length, link density, css classes). It the selects the nodes with the highest score. [1] I ported it to Swift for a personal read later app. [1] https://g…
This is pretty cool. Care to share your Swift port?
Re: Show HN: I made a tool to clean and convert any webpage to Markdown
#87Earlier quoted context omitted.
Thoroughly scraping is challenging, especially in an environment where you don’t have (or want) a JavaScript runtime. For content extraction, I found the approach the Postlight library takes quite neat. It scores individual html nodes based on some heuristics (text length, link density, css classes). It the selects the nodes with the highest score. [1] I ported it to Swift for a personal read later app. [1] https://g…
This is pretty cool. Care to share your Swift port?
Not planning to. It’s my first Swift/iOS project. I neither want to polish it nor maintain it publicly. Happy to share it privately, email is in the bio.
I’m planning on a blog post describing the general approach though!
Re: Show HN: I made a tool to clean and convert any webpage to Markdown
#88I wrote a similar article a while ago: https://blog.platypush.tech/article/Deliver-articles-to-your...
In my case the purpose was to share saved links to my e-reader and used Markdown as an intermediate format through the mercury.js scraping API, but the possibilities are endless.
Re: Show HN: I made a tool to clean and convert any webpage to Markdown
#89Zzzzzzz
Re: Show HN: I made a tool to clean and convert any webpage to Markdown
#90Converting websites to markdown comes with 3 distinct problems: 1. Throughly scraping the content of page (high recall) 2. Dropping all the ads/auxilliary content (high precision) 3. And getting the correct layout/section types (formatting) For #2 and #3 - Trafilatura, Newspaper4k and python-readability based solutions work best out of the box. For #1, any scraping service + selenium is going to do a great job. Could…
Thanks for the links I had no idea those existed.
For my article web scraper (wip) the current steps are:
- Navigate with playwright + adblocker
- Run mozilla's readability on the page
- LLM checks readability output
If check failed
- Trim whole page HTML context
- Convert to markdown with pandoc
- LLM extracts from markdown