Web scraping with GPT-4o: powerful but expensive
1–10 of 177 posts
Re: Web scraping with GPT-4o: powerful but expensive
#2Why are scrapers so popular nowadays?
Re: Web scraping with GPT-4o: powerful but expensive
#3Why are scrapers so popular nowadays?
Because publishers don’t push structured data or APIs enough to satisfy demand for the data.
Re: Web scraping with GPT-4o: powerful but expensive
#4Why are scrapers so popular nowadays?
I'd say scrapers have always been popular, but I imagine they're even more popular nowadays with all the tools (AI but also non-AI) readily available to do cool stuff on a lot of data.
Re: Web scraping with GPT-4o: powerful but expensive
#5Re: Web scraping with GPT-4o: powerful but expensive
#6Is it possible to achieve good results using the open source models for scrapping?
Re: Web scraping with GPT-4o: powerful but expensive
#7To scale such an approach you could have the LLM generate JS to walk the DOM and extract content, caching the JS for each page.
Re: Web scraping with GPT-4o: powerful but expensive
#8Why are scrapers so popular nowadays?
I'd say scrapers have always been popular, but I imagine they're even more popular nowadays with all the tools (AI but also non-AI) readily available to do cool stuff on a lot of data.
Bingo. During the pandemic, I started a project to keep myself busy by trying to scrape stock market ticker data and then do some analysis and make some pretty graphs out of it. I know there are paid services for this, but I wanted to pull it from various websites for free. It took me a couple months to get it right. There are so many corner cases to deal with if the pages aren't exactly the same each time you load them. Now with the help of AI, you can slap together a scraping program in a couple of hours.
Re: Web scraping with GPT-4o: powerful but expensive
#9Earlier quoted context omitted.
Because publishers don’t push structured data or APIs enough to satisfy demand for the data.
Got it, but why is it booming now and often it’s a showcase of llm model? Is there some secret market/ usecase for it?
Scrapers have always been notoriously brittle and prone to breaking completely when pages make even the smallest of structural changes.
Scraping with LLMs bypasses that pitfall because it's more of a summarization task on the whole document, rather than working specifically on a hard-coded document structure to extract specific data.
Re: Web scraping with GPT-4o: powerful but expensive
#10Funnily enough, web scraping was actually the motivating use-case that started my co-founder and I building what is now openpipe.ai. GPT-4 is really good at it, but extremely expensive. But it's actually pretty easy to distill its skill at scraping a specific class of site down to a fine-tuned model that's way cheaper and also really good at scraping that class of site reliably.