21–30 of 177 posts
Why are scrapers so popular nowadays?
The fact that I can’t get my own receipt data from online retailers is unacceptable. I built a CLI Puppeteer scraper to scrape sites like Target, Amazon, Walmart, and Kroger for precisely this reason.
Any website that has my data and doesn’t give me access to it is a great target for scraping.
I'm working on a Chrome extension to do web scraping using OpenAI, and I've been impressed by what ChatGPT can do. It can scrape complicated text/html, and usually returns the correct results. It's very early still but check it out at https://FetchFoxAI.com One of the cool things is that you can scrape non-uniform pages easily. For example I helped someone scrape auto dealer leads from different websites: https://you…
Earlier quoted context omitted.
Got it, but why is it booming now and often it’s a showcase of llm model? Is there some secret market/ usecase for it?
Building scrapers sucks. It's generally not hard because it's conceptually very difficult, or that it requires extremely high level reasoning. It sucks because when someone changes " " to " " your scraper breaks. I just want the bio and it's obvious what to grab, but machines have no nuance. LLMs have enough common sense to be able to deal with these things and they take almost no time to work with. I can throw html…
> I also tried GPT-4o mini but yielded significantly worse results so I just continued my experiments with GPT-4o.
Would be interesting to compare with the other inexpensive top tier models, Claude 3 Haiku and Gemini 1.5 Flash.
Earlier quoted context omitted.
Building scrapers sucks. It's generally not hard because it's conceptually very difficult, or that it requires extremely high level reasoning. It sucks because when someone changes " " to " " your scraper breaks. I just want the bio and it's obvious what to grab, but machines have no nuance. LLMs have enough common sense to be able to deal with these things and they take almost no time to work with. I can throw html…
Another approach is to use a regexp scraper. These are very "loose" and tolerant of changes. For example, RNSAFFN.com uses regular expressions to scrape the Commitments of Traders report from the Commodity Futures Trading Commission every week.
My preferred approach for scraping these days is Playwright Python and CSS selectors to select things from the DOM. Still prone to breakage, but reasonably pleasant to debug using browser DevTools.
I'm working on a Chrome extension to do web scraping using OpenAI, and I've been impressed by what ChatGPT can do. It can scrape complicated text/html, and usually returns the correct results. It's very early still but check it out at https://FetchFoxAI.com One of the cool things is that you can scrape non-uniform pages easily. For example I helped someone scrape auto dealer leads from different websites: https://you…
Cool, would this work on something like instagram? Scraping pages?
Isn't ollama an answer to this? Or is there something inherent to OpenAI that makes it significantly better for web scraping?
I'm working on a Chrome extension to do web scraping using OpenAI, and I've been impressed by what ChatGPT can do. It can scrape complicated text/html, and usually returns the correct results. It's very early still but check it out at https://FetchFoxAI.com One of the cool things is that you can scrape non-uniform pages easily. For example I helped someone scrape auto dealer leads from different websites: https://you…
Cool, would this work on something like instagram? Scraping pages?
I got these results just now: https://fetchfoxai.com/s/UOqL5HtuNe
If you want to do the same scrape, here is the prompt I used: https://imgur.com/XhguCk4
OpenAI recently announced a Batch API [1] which allows you to prepare all prompts and then run them as a batch. This reduces costs as its just 50% the price. Used it a lot with GPT-4o mini in the past and was able to prompt 3000 Items in less than 5min. Could be great for non-realtime applications. [1] https://platform.openai.com/docs/guides/batch