I'm the author of the article, Thanks for posting this article and I appreciate all the feedback received. Indeed, it seems the cost of using the OpenAI API for scrapers at scale is high. However, in my opinion, optimization is key. As some comments suggested, scrapers could employ finely-tuned, smaller models—perhaps distilled from ChatGPT—to achieve similar tasks at a lower cost. One takeaway from this article migh…
But we know if the ever increasing power of compute that problems limited by cost of compute get solved all the time. "Way back in the day" we'd have never really tried to crack passwords on 486's. For example, theses days we're throwing ever more complicated algorithms and requests on the user to ensure the password isn't quickly broken if the cyphertext is stolen.