[deleted]
Web scraping with GPT-4o: powerful but expensive
101–110 of 177 posts
Re: Web scraping with GPT-4o: powerful but expensive
#102[dead]
Re: Web scraping with GPT-4o: powerful but expensive
#103It's very surprising that the author of this post does 99% of the work and writing and then does not go forward for the other 1% downloading ollama (or some other llama.cpp based engine) and testing how some decent local LLM works in this use case. Because maybe a 7B or 30B model will do great in this use case, and that's cheap enough to run: no GPT-4o needed.
Re: Web scraping with GPT-4o: powerful but expensive
#104Isn't ollama an answer to this? Or is there something inherent to OpenAI that makes it significantly better for web scraping?
GPT-4o (and the other top-tier models like Claude 3.5 Sonnet and Gemini 1.4 Pro) is massively more capable than models you can run on your own machine using Ollama - unless you can run something truly monstrous like Llama 3.1 405b, but that's requires 100GBs of GPU RAM which is very expensive.
I understand that if the web scraping activity has some ROI it is perfectly affordable. It is iust for fun it doesn't make sense but the article is already paying for a service and looking for a goal.
Re: Web scraping with GPT-4o: powerful but expensive
#105Can anyone recommend an AI vision web browsing automation framework rather than just scraping? My use case: automate the monthly task of logging into a website and downloading the latest invoice PDF.
Re: Web scraping with GPT-4o: powerful but expensive
#106Same experience here. Been building a classical music database [1] where historical and composer life events are scraped off wikipedia by asking ChatGPT to extract lists of `[{event, year, location}, ...]` from biographies.
- Using chatgpt-mini was the only cheap option, worked well (although I have a feeling it's dumbing down these days) and made it virtually free.
- Just extracting the webpage text from HTML, with `BeautifulSoup(html).text` slashes the number of tokens (but can be risky when dealing with complex tables)
- At some point I needed to scrape ~10,000 pages that have the same format and it was much more efficient speed-wise and price-wise to provide ChatGPT with the HTML once and say "write some python code that extracts data", then apply that code to the 10,000 pages. I'm thinking a very smart GPT-based web parser could do that, with dynamically generated scraping methods.
- Finally because this article mentions tables, Pandas has a very nice feature `from_html("http:/the-website.com")` that will detect and parse all tables on a page. But the article does a good job pointing at websites where the method would fail because the tables don't use ``
Re: Web scraping with GPT-4o: powerful but expensive
#107Is there a "html reducer" out there? I've been considering writing one. If you take a page's source it's going to be 90% garbage tokens -- random JS, ads, unnecessary properties, aggressive nesting for layout rendering, etc. I feel like if you used a dom parser to walk and only keep nodes with text, the html structure and the necessary tag properties (class/id only maybe?) you'd have significant savings. Perhaps the…
author here: I'm working on a follow-up post. Turns out, removing all HTML tags works great and reduces the cost by a huge margin.
Am I crazy or is there no way to “subscribe” to your site? Interested to follow your learnings in this area.
Re: Web scraping with GPT-4o: powerful but expensive
#108Wow, that's one of the most orange tag-rich posts I've ever seen. We're doing a lot of tests with GPT-4o at NewsCatcher. We have to crawl 100k+ news websites and then parse news content. Our rule-based model for extracting data from any article works pretty well, and we never could find a way to improve it with GPT. "Crawling" is much more interesting. We need to know all the places where news articles can be publish…
I’d love to look into this for a hobbyist project I’m working on. Wish you had self signup!
Re: Web scraping with GPT-4o: powerful but expensive
#109I think that LLM costs, even GPT-4o, are probably lower compared to proxy costs usually required for web scraping at scale. The cost of residential/mobile proxies is a few $ per GB. If I were to process cleaned data obtained using 1GB of residential/mobile proxy transfer, I wouldn't pay more for LLM.
Re: Web scraping with GPT-4o: powerful but expensive
#110Same experience here. Been building a classical music database [1] where historical and composer life events are scraped off wikipedia by asking ChatGPT to extract lists of `[{event, year, location}, ...]` from biographies. - Using chatgpt-mini was the only cheap option, worked well (although I have a feeling it's dumbing down these days) and made it virtually free. - Just extracting the webpage text from HTML, with…
If you haven't considered it, you can also use the direct wikitext markup, from which the HTML is derived.
Depending on how you use it, the wikitext may or may not be more ingestible if you're passing it through to an LLM anyway. You may also be able to pare it down a bit by heading/section so that you can reduce it do only sections that are likely to be relevant (eg. "Life and career") type sections.
You can also download full dumps [0] from Wikipedia and query them via SQL to make your life easier if you're processing them.
[0] https://en.wikipedia.org/wiki/Wikipedia:Database_download#Wh...?