Web scraping with GPT-4o: powerful but expensive • Hacker News Board

Re: Web scraping with GPT-4o: powerful but expensive

#131

Post by sebstefan » Tue, Sep 03, 2024, 1:58 PM UTC

melenaboija wrote: post #125

sebstefan wrote: post #119

Earlier quoted context omitted.

The LLM is resistant to website updates that would break normal scraping If you do like the author did and ask it to generate xPaths, you can use it once, use the xPaths it generated for regular scraping, then once it breaks fall back to the LLM to update the xPaths and fall back one more time to alerting a human if the data doesn't start flowing again, or if something breaks further down the pipeline because the dat…

Unless the update is splitting cells. > Turns out, a simple table from Wikipedia (Human development index) breaks the model because rows with repeated values are merged

Nah, still correct :-) that would break the regular scraping as well

>The LLM is resistant to website updates that would break normal scraping

Re: Web scraping with GPT-4o: powerful but expensive

#132

Post by godber » Tue, Sep 03, 2024, 2:00 PM UTC

I would definitely approach this problem by having the LLM write code to scrape the page. That would address the cost and accuracy problems. And also give you testable code.

Re: Web scraping with GPT-4o: powerful but expensive

#133

Post by kanzure » Tue, Sep 03, 2024, 2:00 PM UTC

Instead of directly scraping with GPT-4o, what you could do is have GPT-4o write a script for a simple web scraper and then use a prompt-loop when something breaks or goes wrong.

I have the same opinion about a man and his animals crossing a river on a boat. Instead of spending tokens on trying to solve a word problem, have it create a constraint solver and then run that. Same thing.

Re: Web scraping with GPT-4o: powerful but expensive

#134

Post by sebstefan » Tue, Sep 03, 2024, 2:01 PM UTC

poulpy123 wrote: post #120

namuorg wrote: post #93

For structured content (e.g. lists of items, simple tables), you really don’t need LLMs. I recently built a web scraper to automatically work on any website [0] and built the initial version using AI, but I found that using heuristics based on element attributes and positioning ended up being faster, cheaper, and more accurate (no hallucinations!). For most websites, the non-AI approach works incredibly well so I’d m…

yours is the first one see that allows to scrape by selecting directly what to scrape. I always wondered why there was no tool doing that.

I've seen another website like this that had this feature on hackernews but it was from a retrospective. These websites have the nasty habit of ceasing operations

Re: Web scraping with GPT-4o: powerful but expensive

#135

Post by [deleted] » Tue, Sep 03, 2024, 2:01 PM UTC

sebstefan wrote: post #119

namuorg wrote: post #93

For structured content (e.g. lists of items, simple tables), you really don’t need LLMs. I recently built a web scraper to automatically work on any website [0] and built the initial version using AI, but I found that using heuristics based on element attributes and positioning ended up being faster, cheaper, and more accurate (no hallucinations!). For most websites, the non-AI approach works incredibly well so I’d m…

The LLM is resistant to website updates that would break normal scraping If you do like the author did and ask it to generate xPaths, you can use it once, use the xPaths it generated for regular scraping, then once it breaks fall back to the LLM to update the xPaths and fall back one more time to alerting a human if the data doesn't start flowing again, or if something breaks further down the pipeline because the dat…

[deleted]

Re: Web scraping with GPT-4o: powerful but expensive

#136

Post by kordlessagain » Tue, Sep 03, 2024, 2:06 PM UTC

CSMastermind wrote: post #81

luigi23 wrote: post #2

Why are scrapers so popular nowadays?

There's been a large push to do server-side rendering for web pages which means that companies no longer have a publicly facing API to fetch the data they display on their websites. Parsing the rendered HTML is the only way to extract the data you need.

I've had good success running Playwright screenshots through EasyOCR, so parsing the DOM isn't the only way to do it. Granted, tables end up pretty messy...

Re: Web scraping with GPT-4o: powerful but expensive

#137

Post by [deleted] » Tue, Sep 03, 2024, 2:10 PM UTC

[deleted]

Re: Web scraping with GPT-4o: powerful but expensive

#138

Post by LetsGetTechnicl » Tue, Sep 03, 2024, 2:12 PM UTC

I'm starting to think that LLM's are a solution in need of a problem like how crypto and the blockchain was. Have we not already solved web scraping?

Re: Web scraping with GPT-4o: powerful but expensive

#139

Post by mmasu » Tue, Sep 03, 2024, 2:13 PM UTC

As a poc, we first took a screenshot of the page, cropped it to the part we needed and then passed it to GPT. One of the things we do is compare prices of different suppliers for the same product (i.e. airline tickets), and sometimes need to do it manually. While the approach could look expensive, it is in general cheaper than a real person, and enables the real person to do more meaningful work… so it’s a win-win. I am looking forward to put this in production hopefully

Re: Web scraping with GPT-4o: powerful but expensive

#140

Post by kordlessagain » Tue, Sep 03, 2024, 2:15 PM UTC

albert_e wrote: post #114

Offtopic: What are some good frameworks for webscraping and PDF document processing -- some public and some behind login, some requiring multiple clicks before the sites display relevant data. We need to ingest a wide variety of data sources for one solution. Very few of those sources supply data as API / json.

I have built most of this and have it running on Google Cloud as a service. The framework I built is Open Source. Let me know if you want to discuss: https://mitta.ai