Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Minifying HTML for GPT-4o: Remove all the HTML tags

blancas.io

Post reply on HN
Print view View original

41–50 of 52 posts

Re: Minifying HTML for GPT-4o: Remove all the HTML tags

#41

Post by andybak »

yawnxyz wrote: post #8

I found that reducing html down to markdown using turndown or https://github.com/romansky/dom-to-semantic-markdown works well; if you want the AI to be able to select stuff, give it cheerio or jQuery access to navigate through the html document; if you need to give tags, classes, and ids to the llm, I use an html-to-pug converter like https://www.npmjs.com/package/html2pug which strips a lot of text and cuts costs. I…

But OP did a (admittedly flawed) test. Have you got anything to back up your claim here? We've all got our own hunches but this post was an attempt to test those hypotheses.

Re: Minifying HTML for GPT-4o: Remove all the HTML tags

#42

Post by simplecto »

One of my projects is a virtual agency of multiple LLMs for a variety of back-office services (copywriting, copy-editing, social media, job ads, etc).

We ingest your data wherever you point our crawlers and then clean it for work working in RAGs or chained LLMs.

One library we like a lot is Trafilatura [1]. It does a great job of taking the full HTML page and returning the most semantically relevant parts.

It works well for LLM work as well as generating embeddings for vectors and downstream things.

[1] - https://trafilatura.readthedocs.io/en/latest/

Re: Minifying HTML for GPT-4o: Remove all the HTML tags

#43

Post by ukuina »

simplecto wrote: post #42

One of my projects is a virtual agency of multiple LLMs for a variety of back-office services (copywriting, copy-editing, social media, job ads, etc). We ingest your data wherever you point our crawlers and then clean it for work working in RAGs or chained LLMs. One library we like a lot is Trafilatura [1]. It does a great job of taking the full HTML page and returning the most semantically relevant parts. It works w…

+1 for Trafilatura. Simple, no fuss, and rarely breaks.

I use it nearly hourly for my HN summarizer HackYourNews (https://hackyournews.com).

Re: Minifying HTML for GPT-4o: Remove all the HTML tags

#44

Post by nickpsecurity »

simplecto wrote: post #42

One of my projects is a virtual agency of multiple LLMs for a variety of back-office services (copywriting, copy-editing, social media, job ads, etc). We ingest your data wherever you point our crawlers and then clean it for work working in RAGs or chained LLMs. One library we like a lot is Trafilatura [1]. It does a great job of taking the full HTML page and returning the most semantically relevant parts. It works w…

The paper is good, too, for understanding how it works. The author also mentions many related tools in it.

https://aclanthology.org/2021.acl-demo.15/

Re: Minifying HTML for GPT-4o: Remove all the HTML tags

#45

Post by cpursley »

nprateem wrote: post #38
wis wrote: post #36

Earlier quoted context omitted.

Why do LLMs understand markdown really well? (besides the simple, terse and readable syntax of markdown) They say "LLMs are trained on the web", are the web pages converted from HTML into markdown before being fed into training?

I think it says in the Anthropic docs they use markdown internally (I assume that means were trained on it to a significant extent).

I think Anthropic actually uses xml and OpenAI markdown.

Re: Minifying HTML for GPT-4o: Remove all the HTML tags

#46

Post by bufferout »

sergiotapia wrote: post #25

In Elixir, I select the ` `, then remove all script and style tags. Then extract the text. This results in a kind of innerText you get in browsers, great and light to pass into LLMs. defp extract_inner_text(html) do html |> Floki.parse_document!() |> Floki.find("body") |> Floki.traverse_and_update(fn {tag, _attrs, _children} = _node when tag in ["script", "style"] -> nil node -> node end) |> Floki.text(sep: " ") |> S…

An example of where this approach is problematic: many ecommerce product pages feature embedded json that is used to dynamically update sections of the page.

Re: Minifying HTML for GPT-4o: Remove all the HTML tags

#47

Post by wizzwizz4 »

cal85 wrote: post #18
throwup238 wrote: post #4

I don't think that Mercury Prize table is a representative example because each column has an obviously unique structure that the LLM can key in on: (year) (Single Artist/Album pair) (List of Artist/Album pairs) (image) (citation link) I think a much better test would be something like "List of elements by atomic properties" [1] that has a lot of adjacent numbers in a similar range and overlapping first/last column t…

Good points. But I feel like even with the cities article it could still ‘cheat’ by recognising what the data is supposed to be and filling in the blanks. Does it even need to be real though? What about generating a fake article to use as a test so it can’t possibly recognise the contents? You could even get GPT to generate it, just give it the ‘Largest cities’ HTML and tell it to output identical HTML but with all t…

> You could even get GPT to generate it

This isn't a good idea, if you want a fair test. See https://gwern.net/doc/reinforcement-learning/safe/2023-krako..., specifically https://arxiv.org/abs/1712.02950.

Re: Minifying HTML for GPT-4o: Remove all the HTML tags

#48

Post by furyofantares »

curl-up wrote: post #20
throwup238 wrote: post #4

I don't think that Mercury Prize table is a representative example because each column has an obviously unique structure that the LLM can key in on: (year) (Single Artist/Album pair) (List of Artist/Album pairs) (image) (citation link) I think a much better test would be something like "List of elements by atomic properties" [1] that has a lot of adjacent numbers in a similar range and overlapping first/last column t…

Additionally, using any Wiki page is misleading, as LLMs have seen their format many times during training, and can probably reproduce the original HTML from the stripped version fairly well. Instead, using some random, messy, scattered-with-spam site would be a much more realistic test environment.

Also it can get partial credit on some of these questions without feeding in any data at all.

Re: Minifying HTML for GPT-4o: Remove all the HTML tags

#49

Post by nickpsecurity »

cfcfcf wrote: post #30

I’m curious. Scraping seems to come up a lot lately. What is everyone scraping? And why?

To add to others’ points, we can do two, more things:

1. Pretain models with any legal, scraped content. That includes updating existing models with recent data.

2. Have our own private collection of pages we’ve looked at. Then, we can search them with a local engine.

Re: Minifying HTML for GPT-4o: Remove all the HTML tags

#50

Post by ascorbic »

wis wrote: post #36
bearjaws wrote: post #21

Earlier quoted context omitted.

Seems to be the most common method I've seen, it makes sense given how well LLMs understand markdown.

Why do LLMs understand markdown really well? (besides the simple, terse and readable syntax of markdown) They say "LLMs are trained on the web", are the web pages converted from HTML into markdown before being fed into training?

They're trained on lots of code, and pretty much every public repo has markdown in it, even if it's just the README.
Post reply on HN