We are re-opening coal plants to do this? Every day a bit more disgusted by GenAI stuff
Web scraping with GPT-4o: powerful but expensive
91–100 of 177 posts
Re: Web scraping with GPT-4o: powerful but expensive
#92This is also how we started a while ago. I agree that it's too expensive, hence we're working on making this scalable and cheaper now! We'll soon launch, but here we go! https://expand.ai
I’m curious to know more about your product. Currently, I’m using visualping.io to keep an eye on the website of my local housing community. They share important updates there, and it’s really helpful for me to get an email every few months instead of having to check their site every day.
Re: Web scraping with GPT-4o: powerful but expensive
#93For structured content (e.g. lists of items, simple tables), you really don’t need LLMs.
I recently built a web scraper to automatically work on any website [0] and built the initial version using AI, but I found that using heuristics based on element attributes and positioning ended up being faster, cheaper, and more accurate (no hallucinations!).
For most websites, the non-AI approach works incredibly well so I’d make sure AI is really necessary (e.g. data is unstructured, need to derive or format the output based on the page data) before incorporating it.
Re: Web scraping with GPT-4o: powerful but expensive
#94GPT-4 (and Claude) are definitely the top models out there, but: Llama, even the 8b, is more than capable of handling extraction like this. I've pumped absurd batches through it via vLLM. With serverless GPUs, the cost has been basically nothing.
Can you explain a bit more about what "serverless GPUs" are exactly? Is there a specific cloud provider you're thinking of, e.g. is there a GPU product with AWS? Google gives me SageMaker, which is perhaps what you are referring to?
Re: Web scraping with GPT-4o: powerful but expensive
#95Funnily enough, web scraping was actually the motivating use-case that started my co-founder and I building what is now openpipe.ai. GPT-4 is really good at it, but extremely expensive. But it's actually pretty easy to distill its skill at scraping a specific class of site down to a fine-tuned model that's way cheaper and also really good at scraping that class of site reliably.
Wow, Kyle, you should have mentioned it earlier!
We've been working on this for quite a while. I'll contact you to show how far we've gotten
Re: Web scraping with GPT-4o: powerful but expensive
#96Earlier quoted context omitted.
If you're open to it, I'd love to hear what you think of what we're building at https://browserbase.com/ - you can run a chrome extension on a headless browser so you can do the semantic markdown within the browser, before pulling anything off. We even have an iFrame-able live view of the browser, so your users can get real-time feedback on the XPaths they're generating: https://docs.browserbase.com/features/session-…
This is super neat and I think I've seen your site before :) Do you handle authentication? We have lots of users that want to automate some part of their daily workflow but the pages are often behind a login and/or require a few clicks to reach the desired content. Happy to chat: username@gmail.com
You must get a lot of test emails to that FANTASTIC gmail address. Funny how it might even be worth some decent money.
Re: Web scraping with GPT-4o: powerful but expensive
#97We've been working on AI-automated web scraping at Kadoa[0] and our early experiments were similar to the those in the article. We started when only the expensive and slow GPT-3 was available, which pushed us to develop a cost-effective solution at scale. Here is what we ended up with: - Extraction: We use codegen to generate CSS selectors or XPath extraction code. This is more efficient than using LLMs for every dat…
this! I've been following Kadoa since its very first days. Great team.
Re: Web scraping with GPT-4o: powerful but expensive
#98Earlier quoted context omitted.
I hope some of the opensource inference servers start supporting that endpoint soon. I know vLLM has added some "offline batch mode" support with the same format, they just haven't gotten around to implementing it on the OpenAI endpoint yet.
Do note it can take up to 24 hours or drop requests altogether. But if that’s not an issue for your use case it’s a great cost saving.
What percentage of requests usually get dropped? Is it something miniscule like 1% or are we talking non trivial like 10%
Re: Web scraping with GPT-4o: powerful but expensive
#99Wow, that's one of the most orange tag-rich posts I've ever seen.
We're doing a lot of tests with GPT-4o at NewsCatcher. We have to crawl 100k+ news websites and then parse news content. Our rule-based model for extracting data from any article works pretty well, and we never could find a way to improve it with GPT.
"Crawling" is much more interesting. We need to know all the places where news articles can be published: sometimes 50+ sub-sections.
Interesting hack: I think many projects (including us) can get away with generating the code for extraction since the per-website structure rarely changes.
So, we're looking for LLM to generate a code to parse HTML.
Happy to chat/share our findings if anyone is interested: artem [at] newscatcherapi.com