Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Web scraping with GPT-4o: powerful but expensive

blancas.io

Post reply on HN
Print view View original

111–120 of 177 posts

Re: Web scraping with GPT-4o: powerful but expensive

#111

Post by guwop »

axg11 wrote: post #39
parhamn wrote: post #31

Is there a "html reducer" out there? I've been considering writing one. If you take a page's source it's going to be 90% garbage tokens -- random JS, ads, unnecessary properties, aggressive nesting for layout rendering, etc. I feel like if you used a dom parser to walk and only keep nodes with text, the html structure and the necessary tag properties (class/id only maybe?) you'd have significant savings. Perhaps the…

I wrote an in-house one for Ribbon. If there’s interest, will open source this. It’s amazing how much better our LLM outputs are with the reducer.

sounds amazing!

Re: Web scraping with GPT-4o: powerful but expensive

#112

Post by jeanloolz »

abhgh wrote: post #89

As others have mentioned here you might get better results cheaper (this probably wasn't the point of the article, so just fyi) if you preprocess the html first. I personally have had good results with trafilatura[1], which I don't see mentioned yet. [1] https://trafilatura.readthedocs.io/en/latest/

I second trafilatura greatly. This will save a huge amount of money to just send the text to the LLM. I used it on this recent project (shameless plug): https://github.com/philippe2803/contentmap. It's a simple python library that creates a vector store for any website, using a domain XML sitemap as a starting point. The challenge was that each domain has its own HTML structure, and to create a vector store, we need the actual content, removing HTML tags, etc. Trafilatura basically does that for any url, in just a few lines of code.

Re: Web scraping with GPT-4o: powerful but expensive

#113

Post by spyder »

retrofuturism wrote: post #96
jumploops wrote: post #77

Earlier quoted context omitted.

This is super neat and I think I've seen your site before :) Do you handle authentication? We have lots of users that want to automate some part of their daily workflow but the pages are often behind a login and/or require a few clicks to reach the desired content. Happy to chat: username@gmail.com

You must get a lot of test emails to that FANTASTIC gmail address. Funny how it might even be worth some decent money.

That's not literally his e-mail :D. He means that you have to replace it with his HN username. It would have been better to write it like this: [HN username]@gmail.com

Re: Web scraping with GPT-4o: powerful but expensive

#114

Post by albert_e »

Offtopic:

What are some good frameworks for webscraping and PDF document processing -- some public and some behind login, some requiring multiple clicks before the sites display relevant data.

We need to ingest a wide variety of data sources for one solution. Very few of those sources supply data as API / json.

Re: Web scraping with GPT-4o: powerful but expensive

#115

Post by pbronez »

jumploops wrote: post #41

We've had the best success by first converting the HTML to a simpler format (i.e. markdown) before passing it to the LLM. There are a few ways to do this that we've tried, namely Extractus[0] and dom-to-semantic-markdown[1]. Internally we use Apify[2] and Firecrawl[3] for Magic Loops[4] that run in the cloud, both of which have options for simplifying pages built-in, but for our Chrome Extension we use dom-to-semanti…

First I’ve heard of Semantic Markdown [0]. It appears to be a way to embed RDF data in Markdown documents.

The page I found is labeled “Alpha Draft,” which suggests there isn’t a huge corpus of Semantic Markdown content out there. This might impede LLM’s ability to understand it due to lack of training data. However, it seems sufficiently readable that LLMs could get by pretty well by treating its structured metadata as parathenicals

=====

What is Semantic Markdown?

Semantic Markdown is a plain-text format for writing documents that embed machine-readable data. The documents are easy to author and both human and machine-readable, so that the structured data contained within these documents is available to tools and applications.

Technically speaking, Semantic Markdown is "RDFa Lite for Markdown" and aims at enhancing the HTML generated from Markdown with RDFa Lite attributes.

Design Rationale:

Embed RDFa-like semantic annotation within Markdown

Ability to mix unstructured human-text with machine-readable data in JSON-LD-like lists

Ability to semantically annotate an existing plain Markdown document with semantic annotations

Keep human-readability to a maximum About this document

=====

[0] https://hackmd.io/@sparna/semantic-markdown-draft

Re: Web scraping with GPT-4o: powerful but expensive

#116

Post by [deleted] »

albert_e wrote: post #114

Offtopic: What are some good frameworks for webscraping and PDF document processing -- some public and some behind login, some requiring multiple clicks before the sites display relevant data. We need to ingest a wide variety of data sources for one solution. Very few of those sources supply data as API / json.

[deleted]

Re: Web scraping with GPT-4o: powerful but expensive

#117

Post by zulko »

davidsojevic wrote: post #110
zulko wrote: post #106

Same experience here. Been building a classical music database [1] where historical and composer life events are scraped off wikipedia by asking ChatGPT to extract lists of `[{event, year, location}, ...]` from biographies. - Using chatgpt-mini was the only cheap option, worked well (although I have a feeling it's dumbing down these days) and made it virtually free. - Just extracting the webpage text from HTML, with…

If you haven't considered it, you can also use the direct wikitext markup, from which the HTML is derived. Depending on how you use it, the wikitext may or may not be more ingestible if you're passing it through to an LLM anyway. You may also be able to pare it down a bit by heading/section so that you can reduce it do only sections that are likely to be relevant (eg. "Life and career") type sections. You can also do…

> reduce it do only sections that are likely to be relevant (eg. "Life and career")

True but I also managed to do this from HTML. I tried getting pages wikitext through the API but couldn't find how to.

Just querying the HTML page was less friction and fast enough that I didn't need a dump (although when AI becomes cheap enough, there is probably a lot of things to do from a wikipedia dump!).

One advantage of using online wikipedia instead of a dump is that I have a pipeline on Github Actions where I just enter a composer name and it automagically scrapes the web and adds the composer to the database (takes exactly one minute from the click of the button!).

Re: Web scraping with GPT-4o: powerful but expensive

#118

Post by riiii »

albert_e wrote: post #114

Offtopic: What are some good frameworks for webscraping and PDF document processing -- some public and some behind login, some requiring multiple clicks before the sites display relevant data. We need to ingest a wide variety of data sources for one solution. Very few of those sources supply data as API / json.

I like Crawlee: https://crawlee.dev/

Re: Web scraping with GPT-4o: powerful but expensive

#119

Post by sebstefan »

namuorg wrote: post #93

For structured content (e.g. lists of items, simple tables), you really don’t need LLMs. I recently built a web scraper to automatically work on any website [0] and built the initial version using AI, but I found that using heuristics based on element attributes and positioning ended up being faster, cheaper, and more accurate (no hallucinations!). For most websites, the non-AI approach works incredibly well so I’d m…

The LLM is resistant to website updates that would break normal scraping

If you do like the author did and ask it to generate xPaths, you can use it once, use the xPaths it generated for regular scraping, then once it breaks fall back to the LLM to update the xPaths and fall back one more time to alerting a human if the data doesn't start flowing again, or if something breaks further down the pipeline because the data is in an unexpected format.

Re: Web scraping with GPT-4o: powerful but expensive

#120

Post by poulpy123 »

namuorg wrote: post #93

For structured content (e.g. lists of items, simple tables), you really don’t need LLMs. I recently built a web scraper to automatically work on any website [0] and built the initial version using AI, but I found that using heuristics based on element attributes and positioning ended up being faster, cheaper, and more accurate (no hallucinations!). For most websites, the non-AI approach works incredibly well so I’d m…

yours is the first one see that allows to scrape by selecting directly what to scrape. I always wondered why there was no tool doing that.
Post reply on HN