Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

YouTube-dl has an interpreter for a subset of JavaScript in 870 lines of Python

twitter.com

Post reply on HN
Print view View original

151–160 of 166 posts

Re: YouTube-dl has an interpreter for a subset of JavaScript in 870 lines of Python

#151

Post by ciupicri »

eurasiantiger wrote: post #90
Uptrenda wrote: post #88

Anyone who has ever pulled a website from a script knows the pain that is Javascript. Normally you want to just get some text and work out the API actions but a lot of sites use horribly obfuscated Javascript -- either because that's what modern web development is (lolz) -- or because its part of their 'security.' That means if you want to write browser-based bots properly -- you ought to use a browser. There are spe…

Just npm install puppeteer.

By the way there is also Playwright [1] and it has Python bindings too [2].

[1]: https://playwright.dev/

[2]: https://playwright.dev/python/docs/intro

Re: YouTube-dl has an interpreter for a subset of JavaScript in 870 lines of Python

#152

Post by ciupicri »

Tao3300 wrote: post #133
mkl wrote: post #121

Earlier quoted context omitted.

That is against HN guidelines: "Please submit the original source. If a post reports on something found on another site, submit the latter." - https://news.ycombinator.com/newsguidelines.html

Citing the guidelines is against the guidelines, if not by the letter, in spirit. It's boring and it lacks curiosity. It assumes too much about the sharer. "Can we stop this trend" is a dog whistle for the "HN is getting worse" complaint. Instead we could be considering if we're meant to read the Twitter conversation as well, or sharing a laugh about the link in the tweet author's bio. Or maybe the sharer didn't feel…

How can I read the Twitter conversation when Twitter asks me to log in? :-)

Re: YouTube-dl has an interpreter for a subset of JavaScript in 870 lines of Python

#153

Post by whywhywhywhy »

Arnavion wrote: post #77
jraph wrote: post #74

I do wonder why YouTube does not try harder to make it difficult to do this computation meant to prove you are a legit YouTube web client. Providing an easy-to-find, simple JS function interpretable with 900 lines of Python is like they don't try at all. They might as well do nothing. Or is their goal just to make youtube-dl not 100% reliable? Or to be able to say "look, you are running our code in a way we did not i…

They do make it harder from time to time. In fact yt-dlp's interpreter has been broken for a month or so now and the devs finally gave up and told users to just install PhantomJS (which itself hasn't been updated since 2016 and probably has bugs / vulns of its own, but whatever). https://github.com/yt-dlp/yt-dlp/issues/4635#issuecomment-12...

I mean if this is the direction it’s heading it makes more sense to port yt-dlp to node. It’s already dependent on a scripting language, it may as well be the one YouTube speaks.

Re: YouTube-dl has an interpreter for a subset of JavaScript in 870 lines of Python

#154

Post by atan2 »

This seems to be a pretty small subset of JavaScript, but I personally love small projects like this for educational purposes. Removing the noise and keeping things minimal helps my brain reason about things.

Earlier this year I enrolled in an online class called "Building a Programming Language" taught by Roberto Ierusalimschy (creator of Lua) and Gustavo Pezzi (creator of pikuma.com). We created a toy language interpreter/VM and the final code was around of 1,800 lines of Lua code. Keeping things as simple (and sometimes naive) as possible was definitely the right choice for me to really wrap my head around the basic theory and connect the dots.

Thanks for the link.

Re: YouTube-dl has an interpreter for a subset of JavaScript in 870 lines of Python

#155

Post by Tao3300 »

ciupicri wrote: post #152
Tao3300 wrote: post #133

Earlier quoted context omitted.

Citing the guidelines is against the guidelines, if not by the letter, in spirit. It's boring and it lacks curiosity. It assumes too much about the sharer. "Can we stop this trend" is a dog whistle for the "HN is getting worse" complaint. Instead we could be considering if we're meant to read the Twitter conversation as well, or sharing a laugh about the link in the tweet author's bio. Or maybe the sharer didn't feel…

How can I read the Twitter conversation when Twitter asks me to log in? :-)

cf. paywalls

Re: YouTube-dl has an interpreter for a subset of JavaScript in 870 lines of Python

#156

Post by pwdisswordfish9 »

rkangel wrote: post #94
M30 wrote: post #11

How should a programming noob interpret this? Be impressed at what was achieved here? Be concerned about security implications using the tool? Something else entirely?

This is the compiler writer equivalent of parsing HTML with regex: It is technically wrong - it isn't a sufficiently rich and powerful approach to handle all JS (HTML) that you might throw at it. It'll work for a while until it eventually barfs when you least expect it. EXCEPT that if the inputs you are giving it come from some understood source(s) that aren't likely to change, then a simpler approach to the "all sin…

> some understood source(s) that aren't likely to change

Does that apply to YouTube? Or any of the other hundreds of supported sites?

Re: YouTube-dl has an interpreter for a subset of JavaScript in 870 lines of Python

#157

Post by pwdisswordfish9 »

Supermancho wrote: post #114
tenebrisalietum wrote: post #38

Earlier quoted context omitted.

> How should a programming noob interpret this? The browser is client-facing and everything there is possible to reverse engineer and figure out. So if you design a web-based application, and are depending on client-side Javascript for any security or distribution enforcement, it can be helpful, but can ultimately be unwound and cracked even if obfuscated, etc. > Be impressed at what was achieved here? Yes. Try to do…

Youtube-dl is impressive. This particular hack is not.

youtube-dl as a whole is not particularly impressive either. It’s a big pile of unresolved technical debt, of hacks-upon-hacks and quick-and-dirty temporary solutions just like this one staying there for years.

Re: YouTube-dl has an interpreter for a subset of JavaScript in 870 lines of Python

#158

Post by rkangel »

pwdisswordfish9 wrote: post #156
rkangel wrote: post #94

Earlier quoted context omitted.

This is the compiler writer equivalent of parsing HTML with regex: It is technically wrong - it isn't a sufficiently rich and powerful approach to handle all JS (HTML) that you might throw at it. It'll work for a while until it eventually barfs when you least expect it. EXCEPT that if the inputs you are giving it come from some understood source(s) that aren't likely to change, then a simpler approach to the "all sin…

> some understood source(s) that aren't likely to change Does that apply to YouTube? Or any of the other hundreds of supported sites?

Presumably because it gets tested with those sites and the JS doesn't change that much it can be fixed or adjusted as required.

Re: YouTube-dl has an interpreter for a subset of JavaScript in 870 lines of Python

#159

Post by mid-kid »

lolinder wrote: post #22

To be clear, this is an extremely tiny subset of JS. It looks like they only implemented the features needed to run a very specific function. For example, the only symbol allowed after "new" is "Date", everything else throws an exception. It's still fun that it's there, but it's not as big a deal as it sounds from the tweet.

Yeah, it's essentially used as a javascript expression solver. You can see the full extent of its capabilities in the testsuite: https://github.com/ytdl-org/youtube-dl/blob/master/test/test...

The specific site modules in youtube-dl will take care to extract the bare minimum necessary to solve whatever challenge.

Re: YouTube-dl has an interpreter for a subset of JavaScript in 870 lines of Python

#160

Post by pvillano »

em-bee wrote: post #27
lolinder wrote: post #22

To be clear, this is an extremely tiny subset of JS. It looks like they only implemented the features needed to run a very specific function. For example, the only symbol allowed after "new" is "Date", everything else throws an exception. It's still fun that it's there, but it's not as big a deal as it sounds from the tweet.

if it's going to need much more than that then it probably would make more sense to port the whole application to javascript instead. but then this could be turned into a commandline browser that is able to interpret a whole web-page and save the resulting html structure instead of the source as curl/wget would do.

Eventually, YouTube-dl might have to simulate an entire browser and human user to fool Google. Until then, the usefulness of YouTube-dl is that it's less heavy than a full browser.

I bet someone's already started a YouTube downloader that uses a headless browser

Post reply on HN