Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: I made a tool to clean and convert any webpage to Markdown

markdowndown.vercel.app

Post reply on HN
Print view View original

61–70 of 107 posts

Re: Show HN: I made a tool to clean and convert any webpage to Markdown

#61

Post by genewitch »

cactusfrog wrote: post #31
IanCal wrote: post #27

Earlier quoted context omitted.

Feel free to answer then, how do you do the same functions this does with gpt(3/4) without AI? Edit - This is an excellent use of it, a free text human input capable of doing things like extracting summaries. It does not seem to be used at all for the basic task of extracting content, but for post filtering.

I think “copy from a PDF” could be improved with AI. It’s been 30 years and I still get new lines in the middle of sentences when I try to copy from one.

i've long assumed that is a "feature" of PDF akin to DRM. Making copying text from a PDF makes sense from a publisher's standpoint.

Re: Show HN: I made a tool to clean and convert any webpage to Markdown

#62

Post by jot »

jph00 wrote: post #54
jot wrote: post #46

Great idea to offer image downloads and filtering with GPT! I built a similar tool last year that doesn't have those features: https://url2text.com/ Apologies if the UI is slow - you can see some example output on the homepage. The API it's built on is Urlbox's website screenshot API which performs far better when used directly. You can request markdown along with JS rendered HTML, metadata and screenshot all in one…

Looks nice, but url2text doesn't seem to have an API, and urlbox doesn't seem to have an option to skip the screenshot if you only want the text. And for just the text, it looks to be really expensive.

Thanks!

Sorry it's not clearer but you can skip the screenshot in the Urlbox API if you want to with:

  curl -X POST \
    https://api.urlbox.io/v1/render/sync \
    -H 'Authorization: Bearer YOUR_URLBOX_SECRET' \
    -H 'Content-Type: application/json' \
    -d '
  {
    "url": "example.com",
    "format": "md"
  }
  '
Here's the result of that: https://renders.urlbox.io/urlbox1/renders/5799274d37a8b4e604...

Sorry the pricing isn't a good fit for you. Urlbox has been running for over 11 years. We're bootstrapped and profitable with a team of 3 (plus a few contractors). We're priced to be sustainable so our customers can depend on us in the long term. We automatically give volume discounts as your usage grows.

Re: Show HN: I made a tool to clean and convert any webpage to Markdown

#64

Post by jot »

eshoyuan wrote: post #29

[dead]

Our tool sadly also fails on this: https://url2text.com/u/KYkpBj

The challenge there is that the content is in an iframe.

If you get the URL used for the iframe you can get the content: https://url2text.com/u/kJWaZY

But that's frustrating as it requires two steps.

We might be able to help you get the content from URLs like these in one step. We have quite a bit of power in the Urlbox API that url2text isn't using.

Drop us an email: support@urlbox.com and we'll see what we can do.

Re: Show HN: I made a tool to clean and convert any webpage to Markdown

#65

Post by screye »

Converting websites to markdown comes with 3 distinct problems:

1. Throughly scraping the content of page (high recall)

2. Dropping all the ads/auxilliary content (high precision)

3. And getting the correct layout/section types (formatting)

For #2 and #3 - Trafilatura, Newspaper4k and python-readability based solutions work best out of the box. For #1, any scraping service + selenium is going to do a great job.

Could you elaborate on what your tool does different or better? The area has been stagnant for a while. So curious to hear your learnings.

Re: Show HN: I made a tool to clean and convert any webpage to Markdown

#66

Post by jot »

haddr wrote: post #12
LeonidBugaev wrote: post #3

One of the cases when AI not needed. There is very good working algorithm to extract content from the pages, one of implementations: https://github.com/buriy/python-readability

Some years ago I compared those boilerplate removal tools and I remember that jusText was giving me the best results out of the box (tried readability and few other libraries too). I wonder what is the state of the art today?

This is worth having a look at: https://mixmark-io.github.io/turndown/

With some configuration you can get most of the way there.

Re: Show HN: I made a tool to clean and convert any webpage to Markdown

#67

Post by alsetmusic »

Just tried it on a complex marketing page and it did a great job. Congrats!

I'm curious, if you care to share, what kind of load this places on your host? Is this something that you can keep going for free or will it eventually become non-cost efficient to keep running?

Re: Show HN: I made a tool to clean and convert any webpage to Markdown

#68

Post by nyokodo »

fbdab103 wrote: post #44
LeonidBugaev wrote: post #3

One of the cases when AI not needed. There is very good working algorithm to extract content from the pages, one of implementations: https://github.com/buriy/python-readability

I was honestly expecting it to be mostly black magic, but it looks like the meat of the project is a bunch of (surely hard won) regexes. Nifty.

> I was … expecting it to be mostly black magic, but … the meat of the project is a bunch of … regexes

Wait, regexes are the epitome of black magic. What do you consider as black magic?

Re: Show HN: I made a tool to clean and convert any webpage to Markdown

#70

Post by technics256 »

screye wrote: post #65

Converting websites to markdown comes with 3 distinct problems: 1. Throughly scraping the content of page (high recall) 2. Dropping all the ads/auxilliary content (high precision) 3. And getting the correct layout/section types (formatting) For #2 and #3 - Trafilatura, Newspaper4k and python-readability based solutions work best out of the box. For #1, any scraping service + selenium is going to do a great job. Could…

[dead]
Post reply on HN