101–107 of 107 posts
Pretty cool. I build something very similar - smort.io . Just prepend smort.io/ before any article URL to easily edit, annotate and share it with anyone. Also works on ArXiv papers! This was the Show HN post for Smort - https://news.ycombinator.com/item?id=30673502
Great idea to offer image downloads and filtering with GPT! I built a similar tool last year that doesn't have those features: https://url2text.com/ Apologies if the UI is slow - you can see some example output on the homepage. The API it's built on is Urlbox's website screenshot API which performs far better when used directly. You can request markdown along with JS rendered HTML, metadata and screenshot all in one…
Earlier quoted context omitted.
If the page is rendered using JavaScript then yes you need more. If there were a version of links, elinks, or lynx that executed JS that would be wonderful.
open the page in a browser save it to disk C-s run one of the above commands on the saved file
Converting websites to markdown comes with 3 distinct problems: 1. Throughly scraping the content of page (high recall) 2. Dropping all the ads/auxilliary content (high precision) 3. And getting the correct layout/section types (formatting) For #2 and #3 - Trafilatura, Newspaper4k and python-readability based solutions work best out of the box. For #1, any scraping service + selenium is going to do a great job. Could…
Thoroughly scraping is challenging, especially in an environment where you don’t have (or want) a JavaScript runtime. For content extraction, I found the approach the Postlight library takes quite neat. It scores individual html nodes based on some heuristics (text length, link density, css classes). It the selects the nodes with the highest score. [1] I ported it to Swift for a personal read later app. [1] https://g…