next step: websites add irrelevant text and prompt injections into hidden dom nodes, tags attributes, etc. to prevent llm-based scraping.
Reader-LM: Small Language Models for Cleaning and Converting HTML to Markdown
41–49 of 49 posts
Re: Reader-LM: Small Language Models for Cleaning and Converting HTML to Markdown
#42Not sure about the quality of the model's output. But I really appreciate this little mini-paper they produced. It gives a nice concise description of their goals, benchmarks, dataset preparation, model sizes, challenges and conclusion. And the whole thing is about a 5-10 minute read.
Re: Reader-LM: Small Language Models for Cleaning and Converting HTML to Markdown
#43I never really understand this reasoning of "regex is hard to reason about, so we just use an LLM we custom made instead!" I get it's trendy but reasoning about LLMs is impossible for many devs the idea that this makes it more maintainable is pretty hilarious.
Re: Reader-LM: Small Language Models for Cleaning and Converting HTML to Markdown
#44I never really understand this reasoning of "regex is hard to reason about, so we just use an LLM we custom made instead!" I get it's trendy but reasoning about LLMs is impossible for many devs the idea that this makes it more maintainable is pretty hilarious.
Regex’s require you to understand what the obscure-looking patterns do character by character in a pile of text. Then, across different piles of text. Then, juggling different regex’s.
For a LLM, you can just tune it to produce the right output using examples. Your brain doesn’t have to understand the tedious things it’s doing.
This also replaces a boring, tedious job with one (LLM’s) that’s more interesting. Programmers enjoy those opportunities.
Re: Reader-LM: Small Language Models for Cleaning and Converting HTML to Markdown
#45I never really understand this reasoning of "regex is hard to reason about, so we just use an LLM we custom made instead!" I get it's trendy but reasoning about LLMs is impossible for many devs the idea that this makes it more maintainable is pretty hilarious.
Regex’s require you to understand what the obscure-looking patterns do character by character in a pile of text. Then, across different piles of text. Then, juggling different regex’s. For a LLM, you can just tune it to produce the right output using examples. Your brain doesn’t have to understand the tedious things it’s doing. This also replaces a boring, tedious job with one (LLM’s) that’s more interesting. Program…
In either case you end up with an inscrutable black box into which you pass your html...honestly I'd prefer the black box that runs more efficiently and is intelligible to at least some people (or most, with the help of a big LLM).
Re: Reader-LM: Small Language Models for Cleaning and Converting HTML to Markdown
#46We need one that operates on the visual output
Re: Reader-LM: Small Language Models for Cleaning and Converting HTML to Markdown
#47Pandoc does this very well.
Re: Reader-LM: Small Language Models for Cleaning and Converting HTML to Markdown
#48Earlier quoted context omitted.
Regex’s require you to understand what the obscure-looking patterns do character by character in a pile of text. Then, across different piles of text. Then, juggling different regex’s. For a LLM, you can just tune it to produce the right output using examples. Your brain doesn’t have to understand the tedious things it’s doing. This also replaces a boring, tedious job with one (LLM’s) that’s more interesting. Program…
In either case you end up with an inscrutable black box into which you pass your html...honestly I'd prefer the black box that runs more efficiently and is intelligible to at least some people (or most, with the help of a big LLM).
That is true. One can also document the regex’s and rules well with examples to help visualize it.
I think development time will be the real winner for LLM’s since building the right set of regex’s takes a long time.
I’m not sure which is faster to iterate on when sites change. The regex’s require the human learning one or more regex’s for sites that broke. Then, how they interact with other sites. The LLM might need to be retrained, maybe just see new examples, or might generalize using previous training. Experiments on this would be interesting.
Re: Reader-LM: Small Language Models for Cleaning and Converting HTML to Markdown
#49Earlier quoted context omitted.
In either case you end up with an inscrutable black box into which you pass your html...honestly I'd prefer the black box that runs more efficiently and is intelligible to at least some people (or most, with the help of a big LLM).
That is true. One can also document the regex’s and rules well with examples to help visualize it. I think development time will be the real winner for LLM’s since building the right set of regex’s takes a long time. I’m not sure which is faster to iterate on when sites change. The regex’s require the human learning one or more regex’s for sites that broke. Then, how they interact with other sites. The LLM might need…
Well, even building and commenting the regex is something that LLMs can do pretty well these days. I actually did exactly that, in a different domain: wrote a prompt template that included the current (python-wrapped) regex script and some autogenerated test case results, and a request for a new version of the script. Then passed that to sonnet 3.5 in an unattended loop until all the tests passed. It actually worked.
The secret sauce was knowing what sort of program architecture is suited to that process, and knowing what else should go in the code that would help the LLM get it right.
Which is all to say, use the LLM directly to parse the html, or use an LLM to write the regex to parse the html: both work, but the latter is more efficient.