Vercel!! Watch out for you bill now this is being hugged. Hopefully you are not using like they pester you to do.
What’s the problem with ?
Show HN: I made a tool to clean and convert any webpage to Markdown
91–100 of 107 posts
Re: Show HN: I made a tool to clean and convert any webpage to Markdown
#92I recently built a website using Markdown as the page source, which then uses CommonMark to produce the HTML. I was interested in seeing how different the extracted markdown looked to the original source markdown. It looks identical!
Re: Show HN: I made a tool to clean and convert any webpage to Markdown
#93Converting websites to markdown comes with 3 distinct problems: 1. Throughly scraping the content of page (high recall) 2. Dropping all the ads/auxilliary content (high precision) 3. And getting the correct layout/section types (formatting) For #2 and #3 - Trafilatura, Newspaper4k and python-readability based solutions work best out of the box. For #1, any scraping service + selenium is going to do a great job. Could…
Thanks for the links I had no idea those existed. For my article web scraper (wip) the current steps are: - Navigate with playwright + adblocker - Run mozilla's readability on the page - LLM checks readability output If check failed - Trim whole page HTML context - Convert to markdown with pandoc - LLM extracts from markdown
Mozilla has released Readability as a standalone package so you can avoid spinning up a browser entirely:
https://github.com/mozilla/readability
Re: Show HN: I made a tool to clean and convert any webpage to Markdown
#94Earlier quoted context omitted.
Thanks for the links I had no idea those existed. For my article web scraper (wip) the current steps are: - Navigate with playwright + adblocker - Run mozilla's readability on the page - LLM checks readability output If check failed - Trim whole page HTML context - Convert to markdown with pandoc - LLM extracts from markdown
Mozilla has released Readability as a standalone package so you can avoid spinning up a browser entirely: https://github.com/mozilla/readability
you would still need to run. For js based websites.
Re: Show HN: I made a tool to clean and convert any webpage to Markdown
#95Vercel!! Watch out for you bill now this is being hugged. Hopefully you are not using like they pester you to do.
What’s the problem with ?
Vercel charges for image optimizations/dynamic scaling. People get wrong perception that it should be a free service because the DX is so easy...
Re: Show HN: I made a tool to clean and convert any webpage to Markdown
#96Earlier quoted context omitted.
Thanks for the links I had no idea those existed. For my article web scraper (wip) the current steps are: - Navigate with playwright + adblocker - Run mozilla's readability on the page - LLM checks readability output If check failed - Trim whole page HTML context - Convert to markdown with pandoc - LLM extracts from markdown
Mozilla has released Readability as a standalone package so you can avoid spinning up a browser entirely: https://github.com/mozilla/readability
I still wanted the browser for UBlock Origin and handling sites with heavy JS. I was using the standalone Readability script already but today I ended up dropping it for Trafilatura. It works a lot better.
The inefficiency of using a browser rather than just taking the html doesn't really matter because the limiting factor is the LLM here.
And yes the LLM is essential for getting clean data. None of the existing methods are flexible enough for all cases even if people say "you don't need AI to do this".
Re: Show HN: I made a tool to clean and convert any webpage to Markdown
#97Earlier quoted context omitted.
I was honestly expecting it to be mostly black magic, but it looks like the meat of the project is a bunch of (surely hard won) regexes. Nifty.
> I was … expecting it to be mostly black magic, but … the meat of the project is a bunch of … regexes Wait, regexes are the epitome of black magic. What do you consider as black magic?
Macros? Any situation where code edits other code?
Sure, I could not write a regex engine, but the language itself can be fine if you keep it to straightfoward stuff. Unlike the famous e-mail parsing regex.
Re: Show HN: I made a tool to clean and convert any webpage to Markdown
#98links -dump elinks -dump lynx -dump let me guess, you need more?
If the page is rendered using JavaScript then yes you need more. If there were a version of links, elinks, or lynx that executed JS that would be wonderful.
open the page in a browser
save it to disk C-s
run one of the above commands on the saved file
Re: Show HN: I made a tool to clean and convert any webpage to Markdown
#99Converting websites to markdown comes with 3 distinct problems: 1. Throughly scraping the content of page (high recall) 2. Dropping all the ads/auxilliary content (high precision) 3. And getting the correct layout/section types (formatting) For #2 and #3 - Trafilatura, Newspaper4k and python-readability based solutions work best out of the box. For #1, any scraping service + selenium is going to do a great job. Could…
Thoroughly scraping is challenging, especially in an environment where you don’t have (or want) a JavaScript runtime. For content extraction, I found the approach the Postlight library takes quite neat. It scores individual html nodes based on some heuristics (text length, link density, css classes). It the selects the nodes with the highest score. [1] I ported it to Swift for a personal read later app. [1] https://g…
Care to share the Swift port?
Re: Show HN: I made a tool to clean and convert any webpage to Markdown
#100I've been using gather-cli[0]for this, built by the venerable Brett Terpstra.