Experimental library for scraping websites using OpenAI's GPT API
jamesturk.github.io
Experimental library for scraping websites using OpenAI's GPT API
1–10 of 149 posts
Re: Experimental library for scraping websites using OpenAI's GPT API
#2OpenAI is actively blocking the scraping use case. Does this work around that?
Re: Experimental library for scraping websites using OpenAI's GPT API
#3OpenAI is actively blocking the scraping use case. Does this work around that?
What do you mean by this, and what would be their reason for doing so? I've tested a few prompts for scraping and there have been no problems.
Re: Experimental library for scraping websites using OpenAI's GPT API
#4[dead]
Re: Experimental library for scraping websites using OpenAI's GPT API
#5[deleted]
Re: Experimental library for scraping websites using OpenAI's GPT API
#6Interesting, the though had crossed my mind, and had briefly tested gpt3 years ago for this. H
Have you bench marked it? I might add it too my benchmarking tool for content extraction, https://github.com/Nootka-io/wee-benchmarking-tool.
I want to try sending scrapped screenshots to gpt4 multimodal and see what it can do for IR.
Re: Experimental library for scraping websites using OpenAI's GPT API
#7Great use case!
- LLMs excel at converting unstructured => structured data
- Will become less expensive over time
- When GPT-4 image support launches publicly, would be a cool integration / fallback for cases where the code-based extraction fails to produce desired results
- In theory works on any website regardless of format / tech
Re: Experimental library for scraping websites using OpenAI's GPT API
#8[dead]
All I got is this:
```json { "url": "https://www.3sonsbrewingco.com/menus", "title": "MENU | 3sons", "content": " \r\n\r\nBrewery & Kitchen\r\n\r\nEAT & DRINK\r\n\r\n " } ```
I was hoping for some menu items...
Re: Experimental library for scraping websites using OpenAI's GPT API
#9OpenAI is actively blocking the scraping use case. Does this work around that?
I don't think this is correct at all. It's one of the main use cases for GPT-4 – so long as the scraped data or outputs from their LLMs aren't used to train competing LLMs.
Re: Experimental library for scraping websites using OpenAI's GPT API
#10OpenAI is actively blocking the scraping use case. Does this work around that?
Couldn't find any mention of this, please provide a source.
Their ToS mentions scraping but it pertains to scraping their frontend instead of using their API, which they don't want you to do.
Also - this library requests the HTML by itself [0] and ships it as a prompt but with preset system messages as the instruction [1].
[0] - https://github.com/jamesturk/scrapeghost/blob/main/src/scrap...
[1] - https://github.com/jamesturk/scrapeghost/blob/main/src/scrap...