Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Yt-dlp: Upcoming new requirements for YouTube downloads

github.com

Post reply on HN
Print view View original

501–510 of 635 posts

Re: Yt-dlp: Upcoming new requirements for YouTube downloads

#501

Post by xnx »

Gigachad wrote: post #486
xnx wrote: post #480

Earlier quoted context omitted.

I definitely agree logins will be required for many more sites, but how would the site be able to distinguish humans from bots controlling the browser? Captcha is almost obsolete. ARC AGI is too cumbersome for verifying every time.

Small scale usage at the same level as a normal person would probably go under the radar, but if you try scraping, running multiple accounts or posting any more than you would a normal user it’ll be picked up once they can link all actions to a real person. If you are just asking Siri to load a page for you, that probably gets tolerated. Maybe very sensitive sites will go verified mobile platform only and Apple/Googl…

The next frontier is entire fake personas to login and scrape sites ... which is why government/real-world verification will be required soon.

Re: Yt-dlp: Upcoming new requirements for YouTube downloads

#502

Post by cakealert »

gamer191 wrote: post #482
cakealert wrote: post #417

Why are they using the web target? YouTube has multiple other attack vectors which have no javascript barriers. Plenty of devices have YouTube players which are not being capable of being updated and which must work, exploit those APIs.

"Attack vectors" is a very interesting choice of words. Yt-dlp is literally using a public API for its intended purpose (accessing videos). The only difference is how yt-dlp is delivering the videos to the user. Probably as much of an "attack" as user-agent spoofing or using browser extensions. But to answer your question, no, there aren't any suitable APIs (I've looked into it). They all either require JavaScript (y…

What about the smart TVs? There have to be a lot of them, do all of them run JS?

Also what kind of environments are executing the JS? If Google begins to employ browser fingerprinting that may become relevant.

Re: Yt-dlp: Upcoming new requirements for YouTube downloads

#503

Post by elcritch »

godelski wrote: post #473
TeMPOraL wrote: post #451

Earlier quoted context omitted.

> I mean these numbers are just made up anyways, so why are engineers concerned with them? That's what they're directly or indirectly being graded on. Even if they don't have to show how their work impacted the company's bottom line, their managers or their managers' managers have to, and poop just rolls downhill. > The idea of engineers needing to justify monetary value is just... ill conceived. They should be conce…

> That's what they're directly or indirectly being graded on. I think you'd agree that this should have never been the case. Engineering managers or project managers, sure. But engineers? That's just silly. We need firewalls. One group's primary concern needs to be on the product. Another group's primary concern needs to be on keeping the business alive and profitable. Too much of the former and you fail to prioritiz…

Perhaps it's because I've been in tech for so long, but I can't comprehend PhD candidates not knowing about open source software.

Re: Yt-dlp: Upcoming new requirements for YouTube downloads

#504

Post by wraptile »

mrsilencedogood wrote: post #425
wraptile wrote: post #58

Days of just getting data off the web are coming to an end as everything requires a full browser running thousands of lines of obfuscated js code now. So instead of a website giving me that 1kb json that could be cached now I start a full browser stack and transmit 10 megabytes through 100 requests, messing up your analytics and security profile and everyone's a loser. Yay.

fortunately it is now easier than ever to do small-scale scraping, the kind yt-dlp does. I can literally just go write a script that uses headless firefox + mitmproxy in about an hour or two of fiddling, and as long as I then don't go try to run it from 100 VPS's and scrape their entire website in a huge blast, I can typically archive whatever content I actually care about. Basically no matter what protection mechani…

> fortunately it is now easier than ever to do small-scale scraping, the kind yt-dlp does.

this is absolutely not the case. I've been web scraping since 00s and you could just curl any html or selenium the browser for simple automation but now it's incredibly complex and expensive even with modern tools like playwright and all of the monthly "undetectable" flavors of it. Headless browsers are laughably easy to detect because they leak the fact they are being automated and that they are headless. Not to even mention all of the fingerprinting.

Re: Yt-dlp: Upcoming new requirements for YouTube downloads

#505

Post by int_19h »

ddtaylor wrote: post #301

When I first got with my wife I seemed a bit crazier than I am because I am a media hoarder for 30+ years. I don't have any VHS, DVDs, etc. laying around because I only keep digital copies, but I have pretty decent archives. Nothing important really, just normal stuff and some rare or obscure stuff that disappears over time. My wife was interested in the idea that I was running "Netfix from home" and enjoyed the lack…

I have set up a Plex server and started ripping shows from various streaming providers (using StreamFab mostly) specifically because my wife got frustrated with 1) ads starting to appear even on paid plans and 2) never-ending game of musical chairs where shows move from provider to provider, requiring you to maintain several subscriptions to continue watching. She's not a techie at all, she's just pissed off, and I know she's not the only one.

Let's make sure that when all those people come looking for solutions, they'll find ones that are easy to set up and mostly "just work", at least to the extent this can be done given that content providers are always going to be hostile.

Re: Yt-dlp: Upcoming new requirements for YouTube downloads

#506

Post by int_19h »

TeMPOraL wrote: post #461
goku12 wrote: post #410

Earlier quoted context omitted.

Please remember that an LLM accessing any website isn't the problem here. It's the scraping bots that saturate the server bandwidth (a DoS attack of sorts) to collect data to train the LLMs with. An LLM solving a captcha or an Anubis style proof of work problem isn't a big concern here, because the worst they're going to do with the collected data is to cache them for later analysis and reporting. Unlike the crawlers…

Scraping was a thing before LLMs, there's a whole separate arms race around this for regular competition and "industrial espionage" reasons. I'm not really sure why model training would become a noticeable fraction of scrapping activity - there's only few players on the planet that can afford to train decent LLMs in the first place, and they're not going to re-scrape the content they already have ad infinitum.

> they're not going to re-scrape the content they already have

That's true for static content, but much of it is forums and other places like that where the main value is that new content is constantly generated - but needs to be re-scraped.

Re: Yt-dlp: Upcoming new requirements for YouTube downloads

#507

Post by godelski »

elcritch wrote: post #503
godelski wrote: post #473

Earlier quoted context omitted.

> That's what they're directly or indirectly being graded on. I think you'd agree that this should have never been the case. Engineering managers or project managers, sure. But engineers? That's just silly. We need firewalls. One group's primary concern needs to be on the product. Another group's primary concern needs to be on keeping the business alive and profitable. Too much of the former and you fail to prioritiz…

Perhaps it's because I've been in tech for so long, but I can't comprehend PhD candidates not knowing about open source software.

PhD Economists, not PhD Computer Scientists.

Don't worry, people didn't go completely brain dead lol. And most of the economists know about it but not the scale or how it fits in the larger ecosystem. They really just know it as "there's sometimes tools on GitHub".

Re: Yt-dlp: Upcoming new requirements for YouTube downloads

#508

Post by wraptile »

Gigachad wrote: post #472
xnx wrote: post #391

Earlier quoted context omitted.

It's an arms race. Websites have become stupidly/unnecessarily/hostilely complicated, but AI/LLMs have made it possible (though more expensive) to get whatever useful information exists out of them. Soon, LLMs will be able to complete any Captcha a human can within reasonable time. When that happens, the "analog hole" may be open permanently. If you can point a camera and a microphone at it, the AI will be able to ma…

The future will just be every web session gets tied to a real ID and if the service detects you as a bot you just get blocked by ID.

> The future will just be every web session gets tied to a real ID

This seems like an awful future. We already had this in form of limited ipv4 addresses wher each IP is basically an identity. People started buying up ip addresses and selling them as proxies. So any other form of ID would suffer the same fate unless enforced at government level.

Worst case scenario we have 10,000 people sitting in front of the screens clicking page links because hiring someone to use their "government id" to mindlessly browse the web is the only way to get data of the public web. That's not the future we should want.

Re: Yt-dlp: Upcoming new requirements for YouTube downloads

#509

Post by zenmac »

MangoToupe wrote: post #169
zenmac wrote: post #44

Earlier quoted context omitted.

There are: peertube, odysee, minds, rumble, bitchute web torrent)... It is the same reason why people just can't get off IG. Network effect and in YT case a lot of disk space and bandwidth.

I don’t think network effect matters much if you’re not trying to advertise the content. Organizations can just link to it from their site. I admit I haven’t looked into peertube, and I didn’t think that rumble was any better than YouTube. I don’t recognize the others. Thank you; I’ll resurvey.

yeah there are alternatives for sure, but it takes time to discover them. But many search engine are offering searching of videos. So may just be a good idea to start building a public index of all the videos.

And it is 2025, the HN crowd here can usually just deploy their video to CDN. Many business are also just hosting their own videos.

BTW forgot to mention Odyssey underlying protocol is https://lbry.com

And seems like there are past article about it on HN: https://news.ycombinator.com/item?id=24594663

Re: Yt-dlp: Upcoming new requirements for YouTube downloads

#510

Post by int_19h »

Klonoar wrote: post #419
SpaghettiCthulu wrote: post #289

Earlier quoted context omitted.

I haven't tested this, but in theory running deno with `--v8-flags='--jitless'`[^1][^2] will disable the JIT compiler. [^1]: https://v8.dev/blog/jitless [^2]: https://docs.deno.com/runtime/getting_started/command_line_i...

If the performance drops due to lack of JIT, then GPs comment about effectively useless on mobile might still hold weight.

It's probably a lot more than that. A well-optimized bytecode interpreter is not 100x slower than native code. But also e.g. QuickJS uses refcounting with the occasional tracing to remove cycles, and while it's a simple and reliable approach, it's not fast.
Post reply on HN