Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Experimental library for scraping websites using OpenAI's GPT API

jamesturk.github.io

Post reply on HN
Print view View original

101–110 of 149 posts

Re: Experimental library for scraping websites using OpenAI's GPT API

#101

Post by Buttons840 »

danShumway wrote: post #92

Scraping/structuring data seems to be an area where LLMs are just great. This is a use-case that I think has a lot of potential, it's worth exploring. That being said, I still have to be a stick in the mud and point out that GPT-4 is probably still vulnerable to 3rd-party prompt injection while scraping websites. I've run into people on HN who think that problem is easy to solve. Maybe they're right, maybe they're no…

> GPT-4 is (as far as I know) vulnerable to the Billy Tables attack

GTP4 can't take all the blame for this. If you want a system where GTP can't drop tables, then give it an account that doesn't have permission to drop tables. Build a middleware layer as needed for more complicated situations.

Re: Experimental library for scraping websites using OpenAI's GPT API

#102

Post by welanes »

satvikpendem wrote: post #14

I follow some indie hackers online who are in the scraping space, such as BrowserBear and Scrapingbee, I wonder how they will fare with something like this. The only solace is that this is nondeterministic, but perhaps you can simply ask the API to create Python or JS code that is deterministic, instead. More generally, I wonder how a lot of smaller startups will fare once OpenAI subsumes their product. Those who are…

> perhaps you can simply ask the API to create Python or JS code that is deterministic, instead.

Had a conversation last week with a customer that did exactly that - spent 15 minutes in ChatGPT generating working Scrapy code. Neat to see people solve their own problem so easily but it doesn't yet erode our value.

I run https://simplescraper.io and a lot of value is integrations, scale, proxies, scheduling, UI, not-having-to-maintain-code etc.

More important than that though is time-saved. For many people, 15 minutes wrangling with ChatGPT will always remain less preferable than paying a few dollars and having everything Just Work.

AI is still a little too unreliable at extracting structured data from HTML, but excellent at auxiliary tasks like identifying randomized CSS selectors etc

This will change of course so the opportunity right now is one of arbitrage - use AI to improve your offering before it has a chance to subsume it.

Re: Experimental library for scraping websites using OpenAI's GPT API

#103

Post by acapybara »

pstorm wrote: post #96
rengler33 wrote: post #39

Earlier quoted context omitted.

That sounds really useful, can you provide a link if it's publicly hosted?

It's intimately tied to the rest my repo, but I'll spend some time tonight and try to pull it out into it's own library.

Very interested to see this.

Re: Experimental library for scraping websites using OpenAI's GPT API

#104

Post by stuartaxelowen »

In my experience, the hard part is not extracting data from websites, but observing and implementing the actual structure of the site - e.g. iTunes categories have apps, which have reviews, etc, and making your scraper intelligent enough to make use of that structure to gather the freshest data efficiently.

There is definitely a place for LLMs in solving this problem: in taking over for the human in interpreting the business goals/data to gather along with the available data on the web, but my experiments have shown that this is a significant problem due to limited LLM context length and difficulty distilling messy data. But, very excited to keep pushing, and seeing where things go :)

Note: I build https://www.thoughtvector.io/pointscrape/ to solve very-large-scale web-data gathering problems like these.

Re: Experimental library for scraping websites using OpenAI's GPT API

#105

Post by tomberin »

danShumway wrote: post #92

Scraping/structuring data seems to be an area where LLMs are just great. This is a use-case that I think has a lot of potential, it's worth exploring. That being said, I still have to be a stick in the mud and point out that GPT-4 is probably still vulnerable to 3rd-party prompt injection while scraping websites. I've run into people on HN who think that problem is easy to solve. Maybe they're right, maybe they're no…

This is true of any webscraper though, you need to santitize any content you collect from the web. If a person wanted a scraper to get something different from the browser, they could easily use UA sniffing to do so. (I've seen it this done a few times.)

Asking GPT to create JSON and then validating the JSON is one piece of that process, but before someone deserialized that JSON and executed INSERT statements w/ it, they should do whatever they usually would do to sanitize that input.

Re: Experimental library for scraping websites using OpenAI's GPT API

#106

Post by pharmakom »

timhigins wrote: post #3
pharmakom wrote: post #2

OpenAI is actively blocking the scraping use case. Does this work around that?

What do you mean by this, and what would be their reason for doing so? I've tested a few prompts for scraping and there have been no problems.

Ran into issues asking for JSON output

Re: Experimental library for scraping websites using OpenAI's GPT API

#107

Post by joegibbs »

nghota wrote: post #4

[dead]

I think so. We use GPT for stuff like extracting the author from articles (if they aren't in the Schema.org data or marked up elsewhere), summarising them, extracting relevant tags to link articles together, etc. It's very useful for that kind of information extraction stuff when there's no structure to the data, or the structure is only sometimes followed.

Re: Experimental library for scraping websites using OpenAI's GPT API

#108

Post by Temporary_31337 »

rahimnathwani wrote: post #98
danShumway wrote: post #92

Scraping/structuring data seems to be an area where LLMs are just great. This is a use-case that I think has a lot of potential, it's worth exploring. That being said, I still have to be a stick in the mud and point out that GPT-4 is probably still vulnerable to 3rd-party prompt injection while scraping websites. I've run into people on HN who think that problem is easy to solve. Maybe they're right, maybe they're no…

> Billy Tables Bobby Tables?

The table's been dropped and there was no backup so we'll never find out

Re: Experimental library for scraping websites using OpenAI's GPT API

#109

Post by puglr »

As someone who has been doing the same thing recently, here's how I solved the issue where the page content has to be in the initial HTML.

The first thing I did was fall back to a headless browser. Let it sit for 5 seconds to let the page render, then snatch the innerText.

But 5-10% of sites do a good job of showing you the door for being a robot.

I wanted to try and solve those cases by taking a screenshot of the page and using GPT-4 visual inputs, but when I got access I realized that 1) visual inputs aren't available yet and 2) holy crap is GPT-4 expensive.

So instead what I do is give a screenshot service the url, get back a full-page PNG, then I hand that off to GCP Cloud Vision to OCR it. The OCRed text then gets fed into GPT-3.5 like normal.

Re: Experimental library for scraping websites using OpenAI's GPT API

#110

Post by tomberin »

blueblimp wrote: post #91
readams wrote: post #51

The license for this is pretty hilarious and it's something you should pretty obviously never accept or use under any circumstances.

Yes, it goes beyond even just extensive usage restrictions and restricts _who_ can use it. https://jamesturk.github.io/scrapeghost/LICENSE/#3 It seems, for example, that (by 3.1.12) if you are a person who is involved in the mining of minerals (of any sort), that you are not allowed to use this library, even if you're not using the library for any mining-related purpose.

See the FAQ :)
Post reply on HN