Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Experimental library for scraping websites using OpenAI's GPT API

jamesturk.github.io

Post reply on HN
Print view View original

61–70 of 149 posts

Re: Experimental library for scraping websites using OpenAI's GPT API

#61

Post by Closi »

anonymousDan wrote: post #23
tomberin wrote: post #18

Earlier quoted context omitted.

It seems like he's setting temperature=0 which also means it is deterministic. Anecdotally, I've been playing with it since he posted an earlier link & it does shockingly well on 3.5 and nearly perfectly on 4 for my use cases. (to be clear: I submitted but not the author of the library myself)

Can you elaborate on the temperature parameter? Is this something you can configure in the standard ChatGPT web interface or does it require API access?

GPT basically reads the text you have input, and generates a set of 'likely' next words (technically 'tokens').

So for example, the input:

Bears like to eat ________

GPT may effectively respond with Honey (33% likelihood that honey is the word that follows the statement) and Humans (30% likelihood that humans is the word that follows this statement). GPT is just estimating what word follows next in the sequence based on all it's training data.

With temperature = 0, GPT will always choose "Honey" in the above example.

With temperature != 0, GPT will add some randomness and would occasionally say "Bears like to eat Humans" in the above example.

Strangely a bit of randomness seems to be like adding salt to dinner - just a little bit makes the output taste better for some reason.

Re: Experimental library for scraping websites using OpenAI's GPT API

#63

Post by geepytee »

samwillis wrote: post #21
satvikpendem wrote: post #14

I follow some indie hackers online who are in the scraping space, such as BrowserBear and Scrapingbee, I wonder how they will fare with something like this. The only solace is that this is nondeterministic, but perhaps you can simply ask the API to create Python or JS code that is deterministic, instead. More generally, I wonder how a lot of smaller startups will fare once OpenAI subsumes their product. Those who are…

Scraping using LLMs directly is going to be really quite slow and resource intensive, but obviously quicker to get setup and going. I can see it being useful for quick ad-hock scrapes, but as soon as you need to scrape 10s or 100s thousands of pages it will certainly be better to go the traditional route. Using LLM to write your scrapers though is a perfect use case for them. To put it somewhat in context, the two ty…

I'd invite you to check out https://www.usedouble.com/, we use a combination of LLMs and traditional methods to scrape data and parse the data to answer your questions.

Sure, it may be more resource intensive, but it's not slow by any means. Our users process hundreds of rows in seconds.

Re: Experimental library for scraping websites using OpenAI's GPT API

#65

Post by geepytee »

PUSH_AX wrote: post #12

This was one of the first things I built when I got access to the API, the results ranged from excellent to terrible, it was also non deterministic, meaning I could pipe in the site content twice and the results would be different. Eagerly awaiting my gpt4 access to see if the accuracy improves for this usecase.

You need to set the temperature to 0, and provide as many examples when/where possible to get deterministic results.

For https://www.usedouble.com/ we provide a UI that structures your prompt + examples in a way that achieves deterministic results from web scrapped HTML data.

Re: Experimental library for scraping websites using OpenAI's GPT API

#68

Post by asddubs »

tomberin wrote: post #34
lorey wrote: post #31

Earlier quoted context omitted.

Actual name: Batman Most explicit CSS rules allow you to spot this, implicit rules won't and possibly can't.

:) Agree, but the scraping arms race is way beyond that, if someone doesn't want their page scraped this isn't a threat to them.

i guess the lazy way to prevent this in a foolproof way is to add an ocr somewhere in the pipeline, and use actual images generated from websites. although maybe then you'll get #010101 text on a #000000 background

Re: Experimental library for scraping websites using OpenAI's GPT API

#69

Post by factoidforrest »

Yeah, I built something almost identical in langchain in two days. It can also Google for answers.

Basically in reads through long pages in a loop and cuts out any crap, just returning the main body. And a nice summary too to help with indexing.

Another thing i can do with it is have one LLM go delegate and tell the scraper what to learn from the page, so that I can use a cheaper LLM and avoid taking up token space in the "main" thought process. Classic delegation, really. Like an LLM subprocess. Works great. Just take the output of one and pass it into the output of another so it can say "tell me x information" and then the subprocess will handle it.

Re: Experimental library for scraping websites using OpenAI's GPT API

#70

Post by waboremo »

satvikpendem wrote: post #14

I follow some indie hackers online who are in the scraping space, such as BrowserBear and Scrapingbee, I wonder how they will fare with something like this. The only solace is that this is nondeterministic, but perhaps you can simply ask the API to create Python or JS code that is deterministic, instead. More generally, I wonder how a lot of smaller startups will fare once OpenAI subsumes their product. Those who are…

You're correct, a lot of people are mistaken in this AI gold rush, however they are also misunderstanding how weak their moat actually is and how much AI is going to impact that as well.

Notion does not have a good moat. The increase of AI usage isn't going to strengthen their moat, it's going to weaken it unless they introduce major changes and make it harder for people to transition content away from Notion.

There are a lot of middle men who are going to be shocked to find out how little people care about their layer when openAI can replace it entirely. You know that classic article about how everyone's biggest competitor is a spreadsheet? That spreadsheet just got a little bit smarter.

Post reply on HN