OpenAI recently announced a Batch API [1] which allows you to prepare all prompts and then run them as a batch. This reduces costs as its just 50% the price. Used it a lot with GPT-4o mini in the past and was able to prompt 3000 Items in less than 5min. Could be great for non-realtime applications.
Web scraping with GPT-4o: powerful but expensive
11–20 of 177 posts
Re: Web scraping with GPT-4o: powerful but expensive
#12Earlier quoted context omitted.
I'd say scrapers have always been popular, but I imagine they're even more popular nowadays with all the tools (AI but also non-AI) readily available to do cool stuff on a lot of data.
Bingo. During the pandemic, I started a project to keep myself busy by trying to scrape stock market ticker data and then do some analysis and make some pretty graphs out of it. I know there are paid services for this, but I wanted to pull it from various websites for free. It took me a couple months to get it right. There are so many corner cases to deal with if the pages aren't exactly the same each time you load t…
Was it profitable?
Re: Web scraping with GPT-4o: powerful but expensive
#13Earlier quoted context omitted.
Because publishers don’t push structured data or APIs enough to satisfy demand for the data.
Got it, but why is it booming now and often it’s a showcase of llm model? Is there some secret market/ usecase for it?
Personally I find it's better for archiving as most sites that don't provide a convenient way to save their content directly. Occasionally, I do it just to make a better interface over the data.
Re: Web scraping with GPT-4o: powerful but expensive
#14Isn't ollama an answer to this? Or is there something inherent to OpenAI that makes it significantly better for web scraping?
Re: Web scraping with GPT-4o: powerful but expensive
#15Earlier quoted context omitted.
Because publishers don’t push structured data or APIs enough to satisfy demand for the data.
Got it, but why is it booming now and often it’s a showcase of llm model? Is there some secret market/ usecase for it?
Building scrapers sucks.
It's generally not hard because it's conceptually very difficult, or that it requires extremely high level reasoning.
It sucks because when someone changes "" to "" your scraper breaks. I just want the bio and it's obvious what to grab, but machines have no nuance.
LLMs have enough common sense to be able to deal with these things and they take almost no time to work with. I can throw html at something, with a vague description and pull out structured data with no engineer required, and it'll probably work when the page changes.
There's a huge number of one-off jobs people will do where perfect isn't the goal, and a fast solution + a bit of cleanup is hugely beneficial.
Re: Web scraping with GPT-4o: powerful but expensive
#16Funnily enough, web scraping was actually the motivating use-case that started my co-founder and I building what is now openpipe.ai. GPT-4 is really good at it, but extremely expensive. But it's actually pretty easy to distill its skill at scraping a specific class of site down to a fine-tuned model that's way cheaper and also really good at scraping that class of site reliably.
[deleted]
Re: Web scraping with GPT-4o: powerful but expensive
#17Asking for XPaths is clever!
Plus you can probably use that until it fails (website changes) and then just re scrape it with llm request
Re: Web scraping with GPT-4o: powerful but expensive
#18I'm working on a Chrome extension to do web scraping using OpenAI, and I've been impressed by what ChatGPT can do. It can scrape complicated text/html, and usually returns the correct results.
It's very early still but check it out at https://FetchFoxAI.com
One of the cool things is that you can scrape non-uniform pages easily. For example I helped someone scrape auto dealer leads from different websites: https://youtu.be/QlWX83uHgHs . This would be a lot harder with a "traditional" scraper.
Re: Web scraping with GPT-4o: powerful but expensive
#19I also had good results with structured outputs, scraping news articles for city names from https://lite.cnn.com for the “in the news” list at https://weather.bingo - code here: https://www.jasonthorsness.com/13
I’ve had problems with hallucinations though even for something as simple as city names; also the model often ignores my prompt and returns country names - am thinking of trying a two-stage scrape with one checking the output of the other.
Re: Web scraping with GPT-4o: powerful but expensive
#20Earlier quoted context omitted.
Bingo. During the pandemic, I started a project to keep myself busy by trying to scrape stock market ticker data and then do some analysis and make some pretty graphs out of it. I know there are paid services for this, but I wanted to pull it from various websites for free. It took me a couple months to get it right. There are so many corner cases to deal with if the pages aren't exactly the same each time you load t…
Was it profitable?
I'm sure it was profitable in keeping him busy during the pandemic. Not everything has to derive monetary value, you can do something for experience, fun, kick the tyres, open-source and/or philanthropic avenues.
Besides it's a low margin, heavily capitalized and heavily crowded market you'd be entering and not worth the negative-monetary investment in the short and medium term (unless you wrote AI in the title and we're going to the mooooooon babyh)