Reader-LM: Small Language Models for Cleaning and Converting HTML to Markdown
1–10 of 49 posts
Re: Reader-LM: Small Language Models for Cleaning and Converting HTML to Markdown
#2Maybe I am missing something here, but why would you run "AI" on that task when you go from formal language to formal language?
I don't get the usage of "regex/heuristics" either. Why can that task not be completely handled by a classical algorithm?
Is it about the removal of non-content parts?
Re: Reader-LM: Small Language Models for Cleaning and Converting HTML to Markdown
#3Maybe I am missing something here, but why would you run "AI" on that task when you go from formal language to formal language? I don't get the usage of "regex/heuristics" either. Why can that task not be completely handled by a classical algorithm? Is it about the removal of non-content parts?
It’s informal language that has formal language mixed in. The informal parts determine how the final document should look. So, a simple formal-to-formal translation won’t meet their needs.
Re: Reader-LM: Small Language Models for Cleaning and Converting HTML to Markdown
#4Maybe I am missing something here, but why would you run "AI" on that task when you go from formal language to formal language? I don't get the usage of "regex/heuristics" either. Why can that task not be completely handled by a classical algorithm? Is it about the removal of non-content parts?
There’s html and then there’s… html.
A nicely formatted subset of html is very different from a dom tag soup that is more or less the default nowadays.
Re: Reader-LM: Small Language Models for Cleaning and Converting HTML to Markdown
#5So regex version still beats the LLM solution. There's also the risk of hallucinations. I wonder if they tried to make SML which would rewrite or update the existing regex solution instead of generating the whole content again? This would mean less output tokens, faster inference and output wouldn't contain hallucinations. Although, not sure if small language models are capabable to write regex
Re: Reader-LM: Small Language Models for Cleaning and Converting HTML to Markdown
#6I can say that enough. Small Language Models are the future.
https://www.lycee.ai/blog/why-small-language-models-are-the-...
Re: Reader-LM: Small Language Models for Cleaning and Converting HTML to Markdown
#7Feels surprising that there isn't a modern best-in-class non-LLM alternative for this task. Even in the post, they described that they used a hodgepodge of headless Chrome, readability, lots of regex to create content-only HTML.
Best I can tell, everyone is doing something similar, only differing in the amount of custom situation regex being used.
Re: Reader-LM: Small Language Models for Cleaning and Converting HTML to Markdown
#8When does this SML perform better than hxdelete (or xmlstarlet or whatever) + rdrview + pandoc?
Re: Reader-LM: Small Language Models for Cleaning and Converting HTML to Markdown
#9Feels surprising that there isn't a modern best-in-class non-LLM alternative for this task. Even in the post, they described that they used a hodgepodge of headless Chrome, readability, lots of regex to create content-only HTML. Best I can tell, everyone is doing something similar, only differing in the amount of custom situation regex being used.
How could it possibly be (a better solution) when there are X different ways to do any single thing in html(/css/js)? If you have a website that uses a canvas to showcase the content (think presentation or something like that), where would you even start? People are still discussing whether the semantic web is important; not every page is utf8 encoded, etc. IMHO small LLMS (trained specifically for this) combined with some other (more predictable) techniques are the best solution we are going to get.
Re: Reader-LM: Small Language Models for Cleaning and Converting HTML to Markdown
#10When does this SML perform better than hxdelete (or xmlstarlet or whatever) + rdrview + pandoc?
The answer is in the OP's Reader-LM report:
About their readability-markdown pipeline: "Some users found it too detailed, while others felt it wasn’t detailed enough. There were also reports that the Readability filter removed the wrong content or that Turndown struggled to convert certain parts of the HTML into markdown. Fortunately, many of these issues were successfully resolved by patching the existing pipeline with new regex patterns or heuristics."
To answer their question about the potention of a SML doing this, they see 'room for improvement' - but as their benchmark shows, it's not up to their classic pipeline.
You echo their research question: "instead of patching it with more heuristics and regex (which becomes increasingly difficult to maintain and isn’t multilingual friendly), can we solve this problem end-to-end with a language model?"