[deleted]
Minifying HTML for GPT-4o: Remove all the HTML tags
11–20 of 52 posts
Re: Minifying HTML for GPT-4o: Remove all the HTML tags
#12Is .8 or .9 considered good enough accuracy for something as simple as this?
Yes, because the prompt is simple as well.
Chain of thought or some similar strategies (I hate that they have their own name and like a paper and authors, lol) can help you push that 0.9 to a 0.95-0.99.
Re: Minifying HTML for GPT-4o: Remove all the HTML tags
#13ChatGPT is clearly trained on wikipedia, is there any concern about its knowledge from there polluting the responses? Seems like it would be better to try against data it didn't potentially already know.
Re: Minifying HTML for GPT-4o: Remove all the HTML tags
#14Isn't GPT-4o multimodal? Shouldn't I be able to just feed in an image of the rendered HTML, instead of doing work to strip tags out?
Re: Minifying HTML for GPT-4o: Remove all the HTML tags
#15I roughly came to the same conclusion a few months back and wrote a simple, containerized, open source general purpose scraper for use with GPT using Playwright in C# and TypeScript that's fairly easy to deploy and use with GPT function calling[0]. My observation was that using `document.body.innerText` was sufficient for GPT to "understand" the page and `document.body.innerText` preserves some whitespace in Firefox (and I think Chrome).
I use more or less this code as a starting point for a variety of use cases and it seems to work just fine for my use cases (scraping and processing travel blogs which tend to have pretty consistent layouts/structures).
Some variations can make this better by adding logic to look for the `main` content and ignore `nav` and `footer` (or variants thereof whether using semantic tags or CSS selectors) and taking only the `innerText` from the main container.
Re: Minifying HTML for GPT-4o: Remove all the HTML tags
#16Isn't GPT-4o multimodal? Shouldn't I be able to just feed in an image of the rendered HTML, instead of doing work to strip tags out?
it is theoretically possible, but the results and bandwidth would be worse. sending an image that large would take a lot longer than sending text
Re: Minifying HTML for GPT-4o: Remove all the HTML tags
#17You step back and realize: we are thinking about how to best remove some symbols from documents that not a moment ago we were deciding certainly needed to be in there, all to feed a certain kind of symbol machine which has seen all the symbols before anyway, all so we don't pay as much cents for the symbols we know or think we need.
If I was not a human but some other kind of being suspended above this situation, with no skin in the game so to speak, it would all seem so terribly inefficient... But as fleshy mortal I do understand how we got here.
Re: Minifying HTML for GPT-4o: Remove all the HTML tags
#18I don't think that Mercury Prize table is a representative example because each column has an obviously unique structure that the LLM can key in on: (year) (Single Artist/Album pair) (List of Artist/Album pairs) (image) (citation link) I think a much better test would be something like "List of elements by atomic properties" [1] that has a lot of adjacent numbers in a similar range and overlapping first/last column t…
Good points. But I feel like even with the cities article it could still ‘cheat’ by recognising what the data is supposed to be and filling in the blanks. Does it even need to be real though? What about generating a fake article to use as a test so it can’t possibly recognise the contents? You could even get GPT to generate it, just give it the ‘Largest cities’ HTML and tell it to output identical HTML but with all the names and statistics changed randomly.
Re: Minifying HTML for GPT-4o: Remove all the HTML tags
#19What I do is convert to markdown, that way you still get some semantic structure. Even built an Elixir library for this: https://github.com/agoodway/html2markdown
Re: Minifying HTML for GPT-4o: Remove all the HTML tags
#20I don't think that Mercury Prize table is a representative example because each column has an obviously unique structure that the LLM can key in on: (year) (Single Artist/Album pair) (List of Artist/Album pairs) (image) (citation link) I think a much better test would be something like "List of elements by atomic properties" [1] that has a lot of adjacent numbers in a similar range and overlapping first/last column t…
Additionally, using any Wiki page is misleading, as LLMs have seen their format many times during training, and can probably reproduce the original HTML from the stripped version fairly well.
Instead, using some random, messy, scattered-with-spam site would be a much more realistic test environment.