Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Experimental library for scraping websites using OpenAI's GPT API

jamesturk.github.io

Post reply on HN
Print view View original

141–149 of 149 posts

Re: Experimental library for scraping websites using OpenAI's GPT API

#141

Post by satvikpendem »

dagorenouf wrote: post #135
satvikpendem wrote: post #14

I follow some indie hackers online who are in the scraping space, such as BrowserBear and Scrapingbee, I wonder how they will fare with something like this. The only solace is that this is nondeterministic, but perhaps you can simply ask the API to create Python or JS code that is deterministic, instead. More generally, I wonder how a lot of smaller startups will fare once OpenAI subsumes their product. Those who are…

you’re spot on that A.I could commoditize indie hacking. The problem with many indie hackers is that they just build products to have fun and try to make a quick buck. They take a basic idea and run with it, adding one more competitor to an already jamed market. No serious research or vision. So they get some buzz in the community at launch, then it dies off and they move on to the next idea. Rinse and repeat. Rarely…

I agree Dago (by the way, I enjoy your memes on Twitter). I think too many IHers are just building small features rather than full fledged products. I mean, if they want to make a few k a month, I guess that's alright, but they shouldn't be surprised if they are disrupted easily by competitors and copycats.

A month or two ago, there was some drama (which I'm sure you've seen as well) about an IHer who found a copycat. I looked into it and it didn't seem like a copy at all, yet this person was complaining quite heavily about it. But I mean, it's the fundamental law of business, compete or die. If you can't compete, you're not fit to run your business, and others who can, will.

Re: Experimental library for scraping websites using OpenAI's GPT API

#142

Post by pax »

lorey wrote: post #42
pax wrote: post #36

I'd love a GPT based solution that, provided with similar inputs as ones used by scrapeghost, instead of doing the actual scraping, would rather output a recipe for one of the popular scraping libraries of services - taking care of figuring out the XPaths and the loops for pagination.

Why GPT-based then? There are libraries that do this: You give examples, they generate the rules for you and give you a scraper object that takes any html and returns the scraped data. Mine: https://github.com/lorey/mlscraper Another: https://github.com/alirezamika/autoscraper

Great projects, thank you for the links. On a brief scan neither cover paging/loops - or js frameworks where one would need to use headless browsers and wait for content to load, where a low/lazy code solution might provide the most added value.

Re: Experimental library for scraping websites using OpenAI's GPT API

#143

Post by ec109685 »

hubraumhugo wrote: post #73
satvikpendem wrote: post #14

I follow some indie hackers online who are in the scraping space, such as BrowserBear and Scrapingbee, I wonder how they will fare with something like this. The only solace is that this is nondeterministic, but perhaps you can simply ask the API to create Python or JS code that is deterministic, instead. More generally, I wonder how a lot of smaller startups will fare once OpenAI subsumes their product. Those who are…

Exactly, semantically understanding the website structure is only one challenge of many with web scraping: * Ensuring data accuracy (avoiding hallucination, adapting to website changes, etc.) * Handling large data volumes * Managing proxy infrastructure * Elements of RPA to automate scraping tasks like pagination, login, and form-filling At https://kadoa.com , we are spending a lot of effort solving each of these poi…

Frustrating the only option to learn more is to book a demo and things like the API documentation are dead ends: https://www.kadoa.com/kadoa-api

The landing page does not provide nearly enough information on how it works in practice. Is it automated or is custom code written for each site?

Re: Experimental library for scraping websites using OpenAI's GPT API

#144

Post by puglr »

geysersam wrote: post #140
puglr wrote: post #109

As someone who has been doing the same thing recently, here's how I solved the issue where the page content has to be in the initial HTML. The first thing I did was fall back to a headless browser. Let it sit for 5 seconds to let the page render, then snatch the innerText. But 5-10% of sites do a good job of showing you the door for being a robot. I wanted to try and solve those cases by taking a screenshot of the pa…

I haven't tried this myself yet. But I'm surprised you didn't find it beneficial to pass the raw HTML to the chatbot (potentially after some filtering). Did `innerText` give better results than `innerHTML`? My intuition is that the structure information in the HTML would be useful to extract structured data.

Great question. The problem with the raw HTML was token count. :)

A rather high percentage of pages are far too much for a GPT prompt!

Re: Experimental library for scraping websites using OpenAI's GPT API

#145

Post by elendee »

puglr wrote: post #109

As someone who has been doing the same thing recently, here's how I solved the issue where the page content has to be in the initial HTML. The first thing I did was fall back to a headless browser. Let it sit for 5 seconds to let the page render, then snatch the innerText. But 5-10% of sites do a good job of showing you the door for being a robot. I wanted to try and solve those cases by taking a screenshot of the pa…

why oh why

Re: Experimental library for scraping websites using OpenAI's GPT API

#148

Post by dagorenouf »

satvikpendem wrote: post #141
dagorenouf wrote: post #135

Earlier quoted context omitted.

you’re spot on that A.I could commoditize indie hacking. The problem with many indie hackers is that they just build products to have fun and try to make a quick buck. They take a basic idea and run with it, adding one more competitor to an already jamed market. No serious research or vision. So they get some buzz in the community at launch, then it dies off and they move on to the next idea. Rinse and repeat. Rarely…

I agree Dago (by the way, I enjoy your memes on Twitter). I think too many IHers are just building small features rather than full fledged products. I mean, if they want to make a few k a month, I guess that's alright, but they shouldn't be surprised if they are disrupted easily by competitors and copycats. A month or two ago, there was some drama (which I'm sure you've seen as well) about an IHer who found a copycat…

thanks for the meme appreciation :D.

Yeah I think some people confuse copycats with competitors:

- Copycats who just flat out copy your design / messaging / landing page: that's something to complain about

- Someone doing a product that solves a similar problem but build their own solution and design: that's perfectly normal and acceptable

Re: Experimental library for scraping websites using OpenAI's GPT API

#149

Post by puglr »

elendee wrote: post #145
puglr wrote: post #109

As someone who has been doing the same thing recently, here's how I solved the issue where the page content has to be in the initial HTML. The first thing I did was fall back to a headless browser. Let it sit for 5 seconds to let the page render, then snatch the innerText. But 5-10% of sites do a good job of showing you the door for being a robot. I wanted to try and solve those cases by taking a screenshot of the pa…

why oh why

Heh, mostly as an experiment. I'd done a fair bit of scraping for some personal football apps over the past few years. Was curious about how GPT might be used when starting from first principles, as well as its abilities to solve specific challenges encountered with the traditional approach.
Post reply on HN