Earlier quoted context omitted.
fortunately it is now easier than ever to do small-scale scraping, the kind yt-dlp does. I can literally just go write a script that uses headless firefox + mitmproxy in about an hour or two of fiddling, and as long as I then don't go try to run it from 100 VPS's and scrape their entire website in a huge blast, I can typically archive whatever content I actually care about. Basically no matter what protection mechani…
> fortunately it is now easier than ever to do small-scale scraping, the kind yt-dlp does. this is absolutely not the case. I've been web scraping since 00s and you could just curl any html or selenium the browser for simple automation but now it's incredibly complex and expensive even with modern tools like playwright and all of the monthly "undetectable" flavors of it. Headless browsers are laughably easy to detect…
I made a web scraper in Perl a few years ago. It no longer works because I need a headless browser now or whatever it is called these days.
Web scraping is MUCH WORSE TODAY[1].
[1] I am not yelling, just emphasizing. :)