Minifying HTML for GPT-4o: Remove all the HTML tags

I wonder if this is due to some template engines looking minimalist like that. I think maybe Pug?

https://github.com/pugjs/pug?tab=readme-ov-file#syntax

It is whitespace sensitive though, but essentially looks like that. I doubt this is the only unique template engine like this though.

I don't think that Mercury Prize table is a representative example because each column has an obviously unique structure that the LLM can key in on: (year) (Single Artist/Album pair) (List of Artist/Album pairs) (image) (citation link)

I think a much better test would be something like "List of elements by atomic properties" [1] that has a lot of adjacent numbers in a similar range and overlapping first/last column types. However, the danger with that table might be easy for the LLM to infer just from the element names since they're well known physical constants. The table of counties by population density might be less predictable [2] or list of largest cities [3]

The test should be repeated with every available sorting function too, to see if that causes any new errors.

[1] https://en.wikipedia.org/wiki/List_of_elements_by_atomic_pro...

[2] https://en.wikipedia.org/wiki/List_of_countries_and_dependen...

[3] https://en.wikipedia.org/wiki/List_of_largest_cities#List

thanks a lot for the feedback! you're right, this is much better input data. I'll re-run the code with these tables!

I found that reducing html down to markdown using turndown or https://github.com/romansky/dom-to-semantic-markdown works well;

if you want the AI to be able to select stuff, give it cheerio or jQuery access to navigate through the html document;

if you need to give tags, classes, and ids to the llm, I use an html-to-pug converter like https://www.npmjs.com/package/html2pug which strips a lot of text and cuts costs. I don't think LLMs are particularly trained on pug content though so take this with a grain of salt

Well, when "simply" extracting the core text of an article is a task where most solutions (rule-based, visual, traditional classifiers and LLMs) rarely score above 0.8 in precision on datasets with a variety of websites and / or multilingual pages, I would consider that not too bad.