Experimental library for scraping websites using OpenAI's GPT API

Post by riku_iki » Sat, Mar 25, 2023, 9:43 PM UTC

ZeroGravitas wrote: post #80

rjh29 wrote: post #41

This may finally be a solution for scraping wikipedia and turning it into structured data. (Or do we even need structured data in the post-AI age?) Mediawiki is notorious for being hard to parse: * https://github.com/spencermountain/wtf_wikipedia#ok-first- - why it's hard * https://techblog.wikimedia.org/2022/04/26/what-it-takes-to-p... - an entire article about parsing page TITLES * https://osr.cs.fau.de/wp-content/…

You might find this meets many needs: https://query.wikidata.org/querybuilder/ edit: I tried asking ChatGPT to write SPARQL queries, but the Q123 notation used by Wikidata seems to confuse it. I asked for winners of the Man Booker Prize and it gave me code that was used the Q id for the band Slayer instead of the Booker Prize.

Post by arbuge » Sat, Mar 25, 2023, 9:48 PM UTC

samwillis wrote: post #21

satvikpendem wrote: post #14

I follow some indie hackers online who are in the scraping space, such as BrowserBear and Scrapingbee, I wonder how they will fare with something like this. The only solace is that this is nondeterministic, but perhaps you can simply ask the API to create Python or JS code that is deterministic, instead. More generally, I wonder how a lot of smaller startups will fare once OpenAI subsumes their product. Those who are…

Scraping using LLMs directly is going to be really quite slow and resource intensive, but obviously quicker to get setup and going. I can see it being useful for quick ad-hock scrapes, but as soon as you need to scrape 10s or 100s thousands of pages it will certainly be better to go the traditional route. Using LLM to write your scrapers though is a perfect use case for them. To put it somewhat in context, the two ty…

Post by riku_iki » Sat, Mar 25, 2023, 9:51 PM UTC

ZeroGravitas wrote: post #82

riku_iki wrote: post #81

Earlier quoted context omitted.

its wikidata, not wikipedia, they are two disjoint datasets.

Basically every wikipedia page (across languages) is linked to wikidata, and some infoboxes are generated directly from wikidata, so they're seperate, but overlapping and increasingly so. https://en.wikipedia.org/wiki/Category:Articles_with_infobox... edit: slightly wider scope category pointing to pages using wikidata in different ways: https://en.wikipedia.org/wiki/Category:Wikipedia_categories_...

Post by simonster » Sat, Mar 25, 2023, 9:58 PM UTC

ChaseMeAway wrote: post #75

vhcr wrote: post #43

Earlier quoted context omitted.

Setting temperature to 0 does not make it completely deterministic, from their documentation: > OpenAI models are non-deterministic, meaning that identical inputs can yield different outputs. Setting temperature to 0 will make the outputs mostly deterministic, but a small amount of variability may remain.

My understanding of LLMs is sub-par at best, could someone explain where the randomness comes from in the event that the model temperature is 0? I guess I was imagining that if temperature was 0, and the model was not being continuously trained, the weights wouldn’t change, and the output would be deterministic. Is this a feature of LLMs more generally or has OpenAI more specifically introduced some other degree of r…

Post by tomberin » Sat, Mar 25, 2023, 9:59 PM UTC