Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Experimental library for scraping websites using OpenAI's GPT API

jamesturk.github.io

Post reply on HN
Print view View original

41–50 of 149 posts

Re: Experimental library for scraping websites using OpenAI's GPT API

#41

Post by rjh29 »

This may finally be a solution for scraping wikipedia and turning it into structured data. (Or do we even need structured data in the post-AI age?)

Mediawiki is notorious for being hard to parse:

* https://github.com/spencermountain/wtf_wikipedia#ok-first- - why it's hard

* https://techblog.wikimedia.org/2022/04/26/what-it-takes-to-p... - an entire article about parsing page TITLES

* https://osr.cs.fau.de/wp-content/uploads/2017/09/wikitext-pa... - a paper published about a wikitext parser

Re: Experimental library for scraping websites using OpenAI's GPT API

#42

Post by lorey »

pax wrote: post #36

I'd love a GPT based solution that, provided with similar inputs as ones used by scrapeghost, instead of doing the actual scraping, would rather output a recipe for one of the popular scraping libraries of services - taking care of figuring out the XPaths and the loops for pagination.

Why GPT-based then? There are libraries that do this: You give examples, they generate the rules for you and give you a scraper object that takes any html and returns the scraped data.

Mine: https://github.com/lorey/mlscraper Another: https://github.com/alirezamika/autoscraper

Re: Experimental library for scraping websites using OpenAI's GPT API

#43

Post by vhcr »

tomberin wrote: post #18
PUSH_AX wrote: post #12

This was one of the first things I built when I got access to the API, the results ranged from excellent to terrible, it was also non deterministic, meaning I could pipe in the site content twice and the results would be different. Eagerly awaiting my gpt4 access to see if the accuracy improves for this usecase.

It seems like he's setting temperature=0 which also means it is deterministic. Anecdotally, I've been playing with it since he posted an earlier link & it does shockingly well on 3.5 and nearly perfectly on 4 for my use cases. (to be clear: I submitted but not the author of the library myself)

Setting temperature to 0 does not make it completely deterministic, from their documentation:

> OpenAI models are non-deterministic, meaning that identical inputs can yield different outputs. Setting temperature to 0 will make the outputs mostly deterministic, but a small amount of variability may remain.

Re: Experimental library for scraping websites using OpenAI's GPT API

#44

Post by tomberin »

specproc wrote: post #38
lorey wrote: post #25

Personally, this feels like the direction scraping should move into. From defining how to extract, to defining what to extract. But we're nowhere near that (yet). A few other thoughts from someone who did his best to implement something similar: 1) I'm afraid this is not even close to cost-effective yet. One CSS rule vs. a whole LLM. A first step could be moving the LLM to the client side, reducing costs and latency.…

Yeah, #1 just makes this seem pointless for the time being. The whole point of needing something like this is horizontal scaling. Also not clear from my phone down the pub if inference is needed at each step. That would be slow, no? Even (especially?) if you owned the model.

No inference is needed. IME it can do a single page in ~10s, $0.01/page. Not practical for most use cases, great for a limited few right now.

Re: Experimental library for scraping websites using OpenAI's GPT API

#45

Post by tomberin »

vhcr wrote: post #43
tomberin wrote: post #18

Earlier quoted context omitted.

It seems like he's setting temperature=0 which also means it is deterministic. Anecdotally, I've been playing with it since he posted an earlier link & it does shockingly well on 3.5 and nearly perfectly on 4 for my use cases. (to be clear: I submitted but not the author of the library myself)

Setting temperature to 0 does not make it completely deterministic, from their documentation: > OpenAI models are non-deterministic, meaning that identical inputs can yield different outputs. Setting temperature to 0 will make the outputs mostly deterministic, but a small amount of variability may remain.

TIL, thanks!

Re: Experimental library for scraping websites using OpenAI's GPT API

#46

Post by [deleted] »

satvikpendem wrote: post #14

I follow some indie hackers online who are in the scraping space, such as BrowserBear and Scrapingbee, I wonder how they will fare with something like this. The only solace is that this is nondeterministic, but perhaps you can simply ask the API to create Python or JS code that is deterministic, instead. More generally, I wonder how a lot of smaller startups will fare once OpenAI subsumes their product. Those who are…

[deleted]

Re: Experimental library for scraping websites using OpenAI's GPT API

#47

Post by w3454 »

rjh29 wrote: post #41

This may finally be a solution for scraping wikipedia and turning it into structured data. (Or do we even need structured data in the post-AI age?) Mediawiki is notorious for being hard to parse: * https://github.com/spencermountain/wtf_wikipedia#ok-first- - why it's hard * https://techblog.wikimedia.org/2022/04/26/what-it-takes-to-p... - an entire article about parsing page TITLES * https://osr.cs.fau.de/wp-content/…

What's wild is that the markup for Wikipedia is not that crazy compared to Wiktionary, which has a different format for every single language.

Re: Experimental library for scraping websites using OpenAI's GPT API

#48

Post by mateuszbuda »

satvikpendem wrote: post #14

I follow some indie hackers online who are in the scraping space, such as BrowserBear and Scrapingbee, I wonder how they will fare with something like this. The only solace is that this is nondeterministic, but perhaps you can simply ask the API to create Python or JS code that is deterministic, instead. More generally, I wonder how a lot of smaller startups will fare once OpenAI subsumes their product. Those who are…

In this particular case, GPT can help you mostly with parsing the website but not with the most challenging part of web scraping which is not getting blocked. In this case, you still need a proxy. The value from using web scraping APIs is access to a proxy pool via REST API.

Re: Experimental library for scraping websites using OpenAI's GPT API

#49

Post by passion__desire »

fnordpiglet wrote: post #17
transitivebs wrote: post #7

Great use case! - LLMs excel at converting unstructured => structured data - Will become less expensive over time - When GPT-4 image support launches publicly, would be a cool integration / fallback for cases where the code-based extraction fails to produce desired results - In theory works on any website regardless of format / tech

What I think is super compelling is other AI techniques excel at reasoning about structured data and making complex inferences. Using a feedback cycle ensemble model between LLMs and other techniques I think is how the true power of LLMs will be unlocked. For instance many techniques can reason about stuff expressed in RDF, and gpt4 does a pretty good job changing text blobs like web pages into decent and well formed…

I would love for multimodal models to learn generative art process. e.g. processing or houdini, etc. Being able to map programs in those languages to how they look visually would be a great multiplier for generative artists. Then exploring the latent space through text.

Re: Experimental library for scraping websites using OpenAI's GPT API

#50

Post by sebzim4500 »

tomberin wrote: post #34
lorey wrote: post #31

Earlier quoted context omitted.

Actual name: Batman Most explicit CSS rules allow you to spot this, implicit rules won't and possibly can't.

:) Agree, but the scraping arms race is way beyond that, if someone doesn't want their page scraped this isn't a threat to them.

Has it? Can you give me an example of a site that is hard to scrape by a motivated attacker?

I'm curious, because I've seen stuff like the above but of course it only fools a few off the shelf tools, it does nothing if the attacker is willing to write a few lines of node.js

Post reply on HN