I've been looking for this! My method requires too many steps. I look forward to seeing if this improves my results. Thanks!
Show HN: I made a tool to clean and convert any webpage to Markdown
51–60 of 107 posts
Re: Show HN: I made a tool to clean and convert any webpage to Markdown
#52One of the cases when AI not needed. There is very good working algorithm to extract content from the pages, one of implementations: https://github.com/buriy/python-readability
Last time I tried readability it worked well with articles but struggled with other kinds of pages. Took away far more content than I wanted it to.
Re: Show HN: I made a tool to clean and convert any webpage to Markdown
#53Tried it earlier today, but it was hugged to death. Tried it now, but it only gave me the contents of the cookie-wall, not the page I was looking for.
Tried on another page of the same site, then it only gave me the last article on a 6-article page, some weird things going on.
Re: Show HN: I made a tool to clean and convert any webpage to Markdown
#54Great idea to offer image downloads and filtering with GPT! I built a similar tool last year that doesn't have those features: https://url2text.com/ Apologies if the UI is slow - you can see some example output on the homepage. The API it's built on is Urlbox's website screenshot API which performs far better when used directly. You can request markdown along with JS rendered HTML, metadata and screenshot all in one…
Looks nice, but url2text doesn't seem to have an API, and urlbox doesn't seem to have an option to skip the screenshot if you only want the text. And for just the text, it looks to be really expensive.
Re: Show HN: I made a tool to clean and convert any webpage to Markdown
#55Re: Show HN: I made a tool to clean and convert any webpage to Markdown
#56I use markdownload extension for Firefox. Seems to work pretty ok.
Re: Show HN: I made a tool to clean and convert any webpage to Markdown
#57Nice! I did something similar a while back, but just for substack :)
One example --
URL: https://newsletter.pragmaticengineer.com/p/zirp-software-eng...
Sanitized output: https://substack-ai.vercel.app/u/pragmaticengineer/p/zirp-so...
Raw markdown: https://substack-ai.vercel.app/api/users/pragmaticengineer/p...
(Would be happy to open source it if anyone cares!)
Re: Show HN: I made a tool to clean and convert any webpage to Markdown
#58Is this open sourced anywhere by any chance? Are you using GPT to do the conversion, or just doing it yourself by ways of HTML -> Markdown substitutions?
Re: Show HN: I made a tool to clean and convert any webpage to Markdown
#59Vercel!! Watch out for you bill now this is being hugged. Hopefully you are not using like they pester you to do.
Re: Show HN: I made a tool to clean and convert any webpage to Markdown
#60Nice! I did something similar a while back, but just for substack :) One example -- URL: https://newsletter.pragmaticengineer.com/p/zirp-software-eng... Sanitized output: https://substack-ai.vercel.app/u/pragmaticengineer/p/zirp-so... Raw markdown: https://substack-ai.vercel.app/api/users/pragmaticengineer/p... (Would be happy to open source it if anyone cares!)
I'd be interested in seeing the code, I love working on tools like this