Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Web scraping with GPT-4o: powerful but expensive

blancas.io

Post reply on HN
Print view View original

151–160 of 177 posts

Re: Web scraping with GPT-4o: powerful but expensive

#151

Post by peab »

jumploops wrote: post #77
pkiv wrote: post #59

Earlier quoted context omitted.

If you're open to it, I'd love to hear what you think of what we're building at https://browserbase.com/ - you can run a chrome extension on a headless browser so you can do the semantic markdown within the browser, before pulling anything off. We even have an iFrame-able live view of the browser, so your users can get real-time feedback on the XPaths they're generating: https://docs.browserbase.com/features/session-…

This is super neat and I think I've seen your site before :) Do you handle authentication? We have lots of users that want to automate some part of their daily workflow but the pages are often behind a login and/or require a few clicks to reach the desired content. Happy to chat: username@gmail.com

I'm also curious about this! I've been learning about scraping, but I've had a hard time finding good info about how to deal with user auth effectively.

Re: Web scraping with GPT-4o: powerful but expensive

#152

Post by nickpsecurity »

parhamn wrote: post #31

Is there a "html reducer" out there? I've been considering writing one. If you take a page's source it's going to be 90% garbage tokens -- random JS, ads, unnecessary properties, aggressive nesting for layout rendering, etc. I feel like if you used a dom parser to walk and only keep nodes with text, the html structure and the necessary tag properties (class/id only maybe?) you'd have significant savings. Perhaps the…

That’s easy to do with BeautifulSoup in Python. Look up tutorials on that. Use it on non-essential tags. That will at least work when the content is in HTML rather than procedurally generated (eg JavaScript).

Re: Web scraping with GPT-4o: powerful but expensive

#153

Post by mistercow »

nickpsecurity wrote: post #150
mistercow wrote: post #148

Earlier quoted context omitted.

Have you compared markdown to just stripping the HTML down (strip tag attributes, unwrap links, remove obvious non-displaying elements)? My experience has been that performance is pretty similar to markdown, and it’s an easier transformation with fewer edge cases.

That’s what I’ve done for quite a few [non-LLM] applications. The remaining problem is that HTML is verbose vs other formats. That has a higher, per-token cost. So, maybe stripping followed by substituting HTML tags with a compressed notation.

Yeah it’s slightly more token heavy, although not as much as it seems like at first glance. Most opening tags are 2-3 tokens, and most closing tags are 3-4. Since tags are generally a pretty small fraction of the text, it typically doesn’t make a huge difference IME, but it obviously depends on the particular content.

Re: Web scraping with GPT-4o: powerful but expensive

#154

Post by fzysingularity »

kordlessagain wrote: post #136
CSMastermind wrote: post #81

Earlier quoted context omitted.

There's been a large push to do server-side rendering for web pages which means that companies no longer have a publicly facing API to fetch the data they display on their websites. Parsing the rendered HTML is the only way to extract the data you need.

I've had good success running Playwright screenshots through EasyOCR, so parsing the DOM isn't the only way to do it. Granted, tables end up pretty messy...

We've been doing something simliar for VLM Run [1]. A lot of websites that have obfuscated HTML / JS or rendered charts / tables tend to be hard to parse with the DOM. Taking screenshots are definitely more reliable and future-proof as these webpages are built for humans to interact with.

That said, the costs can be high as the OP says, but we're building cheaper and more specialized models for web screenshot -> JSON parsing.

Also, it turns out you can do a lot more than just web-scraping [2].

[1] https://vlm.run

[2] https://docs.vlm.run/introduction

Re: Web scraping with GPT-4o: powerful but expensive

#156

Post by mthoms »

bilater wrote: post #155

Not sure why author didn't use 4o-mini. 4o for reasoning but things like parsing/summarizing can be done by cheaper models with little loss in quality.

> Note: I also tried GPT-4o mini but yielded significantly worse results so I just continued my experiments with GPT-4o.

Re: Web scraping with GPT-4o: powerful but expensive

#157

Post by defgeneric »

nickpsecurity wrote: post #150
mistercow wrote: post #148

Earlier quoted context omitted.

Have you compared markdown to just stripping the HTML down (strip tag attributes, unwrap links, remove obvious non-displaying elements)? My experience has been that performance is pretty similar to markdown, and it’s an easier transformation with fewer edge cases.

That’s what I’ve done for quite a few [non-LLM] applications. The remaining problem is that HTML is verbose vs other formats. That has a higher, per-token cost. So, maybe stripping followed by substituting HTML tags with a compressed notation.

I've tried this and found it doesn't make much difference. The idea was to somehow preserve the document structure while reducing the token count, so you do things like strip all styles, etc. until you have something like a structure of divs, then reduce that. But I found no performance gain in terms of output. It seems whatever structure of the document is left over after doing the reduction has little semantic meaning that can't be conveyed by spaces or newlines. Even when using something like html2markdown, it doesn't perform much better. So in a sense the LLM is "too good", and all you really need to worry about is reducing the token count.

Re: Web scraping with GPT-4o: powerful but expensive

#158

Post by devoutsalsa »

antirez wrote: post #103

It's very surprising that the author of this post does 99% of the work and writing and then does not go forward for the other 1% downloading ollama (or some other llama.cpp based engine) and testing how some decent local LLM works in this use case. Because maybe a 7B or 30B model will do great in this use case, and that's cheap enough to run: no GPT-4o needed.

Not OP, but thanks for the suggestion. I’m starting to play around with LLMs and will explore locally hosted versions.

Re: Web scraping with GPT-4o: powerful but expensive

#159

Post by LetsGetTechnicl »

gallerdude wrote: post #141
LetsGetTechnicl wrote: post #138

I'm starting to think that LLM's are a solution in need of a problem like how crypto and the blockchain was. Have we not already solved web scraping?

New technologies can solve problems better. "I'm starting to think computers are a solution in the need of a problem. Have we not already solved doing math?"

Does it do it better? It also uses a ton more electricity so even if it's better, is it worth the cost?

Re: Web scraping with GPT-4o: powerful but expensive

#160

Post by CalRobert »

axg11 wrote: post #39
parhamn wrote: post #31

Is there a "html reducer" out there? I've been considering writing one. If you take a page's source it's going to be 90% garbage tokens -- random JS, ads, unnecessary properties, aggressive nesting for layout rendering, etc. I feel like if you used a dom parser to walk and only keep nodes with text, the html structure and the necessary tag properties (class/id only maybe?) you'd have significant savings. Perhaps the…

I wrote an in-house one for Ribbon. If there’s interest, will open source this. It’s amazing how much better our LLM outputs are with the reducer.

Add my voice to the chorus asking for this!
Post reply on HN