Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

YouTube-dl has an interpreter for a subset of JavaScript in 870 lines of Python

twitter.com

Post reply on HN
Print view View original

81–90 of 166 posts

Re: YouTube-dl has an interpreter for a subset of JavaScript in 870 lines of Python

#81

Post by geysersam »

jraph wrote: post #67
sebzim4500 wrote: post #61

Earlier quoted context omitted.

I'm trying to get the thread model here. Is the concern that Youtube will inject JS into the payload which tries to break out of the youtuble-dl js sandbox using some zero day in whatever js engine they would use instead?

Let's say they end up using Node. Node has a quite complete standard library that lets you access files and everything. Now if they do it right and only embed some bare JS interpreter, it's still way harder to audit than these < 900 lines, for which it is quite easy to convince oneself that the interpreted script cannot do much.

Nowadays they could probably use Deno. Without permissions it doesn't allow network or file access etc.

Re: YouTube-dl has an interpreter for a subset of JavaScript in 870 lines of Python

#82

Post by tsukikage »

btdmaster wrote: post #76
tsukikage wrote: post #50

Earlier quoted context omitted.

> you also think billionares should be taxed more That's... quite a response in defense of a tool intended for breaching TOS and performing copyright infringement. Can you clarify exactly who it is and isn't OK to steal from, again? I'm struggling here.

> As a matter of policy (as well as legality), youtube-dl does not include support for services that specialize in infringing copyright. As a rule of thumb, if you cannot easily find a video that the service is quite obviously allowed to distribute (i.e. that has been uploaded by the creator, the creator's distributor, or is published under a free license), the service is probably unfit for inclusion to youtube-dl. D…

Copyright law only permits making copies of artistic works when you have license to do so. Youtube only permits use of content it serves in the specific situations described in its terms of service. All other use is prohibited.

You can see the terms of service here:

https://www.youtube.com/static?gl=GB&template=terms

In particular, the first three points in the "permissions and restrictions" section explicitly prohibit tools like youtube-dl. I've pasted these below:

    The following restrictions apply to your use of the Service. You are not allowed to:

    1. access, reproduce, download, distribute, transmit, broadcast, display, sell, license, alter, modify or otherwise use any part of the Service or any Content except: (a) as specifically permitted by the Service;  (b) with prior written permission from YouTube and, if applicable, the respective rights holders; or (c) as permitted by applicable law;
    2. circumvent, disable, fraudulently engage, or otherwise interfere with the Service (or attempt to do any of these things), including security-related features or features that: (a) prevent or restrict the copying or other use of Content; or (b) limit the use of the Service or Content;
    3. access the Service using any automated means (such as robots, botnets or scrapers) except: (a) in the case of public search engines, in accordance with YouTube’s robots.txt file; (b) with YouTube’s prior written permission; or (c) as permitted by applicable law;
As a convenient figleaf, it is also possible to use youtube-dl for some purposes that are not dubious. Of the people I know who use the tool, none of them do that.

Re: YouTube-dl has an interpreter for a subset of JavaScript in 870 lines of Python

#83

Post by userbinator »

randyrand wrote: post #34
sylware wrote: post #19

Nowadays "javascript" refers to the scriptable, grotesquely and absurdely complex and massive web engines, aka google financed blink and geeko, then apple financed webkit, that with their SDK. The currently obfuscated javascript media players will try to break yt-dlp by leveraging the complexity and size of those scripted web engines. They will make them out of reach to small teamns or individuals and it is even "bet…

Chrome and Safari both have open source JS engines…

That's beside the point. Open-source is not useful to the smaller players if it is too complex to comprehend and constantly churned.

Re: YouTube-dl has an interpreter for a subset of JavaScript in 870 lines of Python

#84

Post by _pn3l »

blast wrote: post #70
esprehn wrote: post #42

This isn't really JS, it's a purpose built evaluator that's only for evaluating a particular script on YouTube, assuming a huge list of things are true about how YouTube JS is written. Ex. Its got a hard coded list of methods for String, and it doesn't respect prototypes. It only supports creating Date instances, and won't work if you override the global Date. It parses with regexes and implements all operators with…

I suppose this means it would be easy for YouTube to fuck with youtube-dl simply by throwing in more features of JS?

Cat, meet mouse.

Re: YouTube-dl has an interpreter for a subset of JavaScript in 870 lines of Python

#85

Post by olliej »

esprehn wrote: post #43
olliej wrote: post #21

This is super cool. Some of the stuff is kind of questionable to me in the sense that I could believe you could probably make some kind of sufficiently wonky JS that this would do the "wrong" thing. But it's super cool that they are able to do this as I think it shows that claims of JS complexity based on the size of JS engines is overlooking just how much of that size/complexity comes from the "make it fast" drive v…

This doesn't actually implement any of the JS language though, it just reuses all of python's semantics and hard coded a tiny list of ex. String methods I also assume you mean mainstream JS engine, but Duktape, JerryScript and QuickJS are all C APIs. They probably could have used ex. https://github.com/PetterS/quickjs instead of the hacks in the OP linked file.

Ah, I only briefly scanned the implementation, and it looked like it was doing actual work - is it mostly string replacing to get approximate python equivalent syntax? Regardless that's disappointing.

You are correct though that I was only thinking of the big engines - bias on my part alas.

For your suggested alternate engines, JerryScript and QuickJS seem more complete than Duktape but I can't quite work out the GC strategy of JerryScript. Bellard says QuickJS has a cycle detector but I'm generally dubious of them based on prior experience.

If I was shipping software that had to actually include a JS engine, if perf was not an issue I would probably use JerryScript or QuickJS as binary size I think would be a more critical component.

Re: YouTube-dl has an interpreter for a subset of JavaScript in 870 lines of Python

#86

Post by lewisl9029 »

Another really cool JS dialect I recently learned about is njs from the nginx team: https://github.com/nginx/njs

This video goes into some of the design and tradeoffs: https://www.youtube.com/watch?v=Jc_L6UffFOs

TL;DW: they optimized for fast creation/destruction of low-footprint VMs with no JIT or garbage collection.

Re: YouTube-dl has an interpreter for a subset of JavaScript in 870 lines of Python

#87

Post by Cthulhu_ »

jraph wrote: post #74

I do wonder why YouTube does not try harder to make it difficult to do this computation meant to prove you are a legit YouTube web client. Providing an easy-to-find, simple JS function interpretable with 900 lines of Python is like they don't try at all. They might as well do nothing. Or is their goal just to make youtube-dl not 100% reliable? Or to be able to say "look, you are running our code in a way we did not i…

I'm guessing the amount of people using it is low enough to not bother with mitigation. Then again, there's a LOT of YT videos that take clips from other videos (which in most cases falls under fair use), which I can imagine would use this tool.

Re: YouTube-dl has an interpreter for a subset of JavaScript in 870 lines of Python

#88

Post by Uptrenda »

Anyone who has ever pulled a website from a script knows the pain that is Javascript. Normally you want to just get some text and work out the API actions but a lot of sites use horribly obfuscated Javascript -- either because that's what modern web development is (lolz) -- or because its part of their 'security.' That means if you want to write browser-based bots properly -- you ought to use a browser. There are special browsers that run 'headlessly' or are designed mostly for bot use. Like https://www.selenium.dev/ which plugs into a few different 'browser engines.'

But now you have another problem. Your simple script goes from being small, simple, self-contained, and elegant gem, to requiring a full browser, specialized drivers, and/or daemons running just to work. If you're using something like Python you just frankly don't have very good packaging. So it's hard to string together all that into a solution and have it magically work for everyone. What YouTube-dl have done is good engineering. Even though it's not a full JS interpreter: they've kept their software lean, self-contained, and easier to use.

Re: YouTube-dl has an interpreter for a subset of JavaScript in 870 lines of Python

#89

Post by jiggawatts »

jraph wrote: post #55
esprehn wrote: post #42

This isn't really JS, it's a purpose built evaluator that's only for evaluating a particular script on YouTube, assuming a huge list of things are true about how YouTube JS is written. Ex. Its got a hard coded list of methods for String, and it doesn't respect prototypes. It only supports creating Date instances, and won't work if you override the global Date. It parses with regexes and implements all operators with…

And as a user of youtube-dl, I'm quite happy about this. This probably allows a very safe, restricted "subset" of JS. Way better than using a full JS engine. 900 lines is still small and manageable.

That’s the exact same logic I hear from developers who say things like:

Why do I need a full XML parser when I can just extract what I need with regex?

And:

All that RPC IDL stuff is overcomplicated, REST is so much easier because I can just write the client by hand.

Re: YouTube-dl has an interpreter for a subset of JavaScript in 870 lines of Python

#90

Post by eurasiantiger »

Uptrenda wrote: post #88

Anyone who has ever pulled a website from a script knows the pain that is Javascript. Normally you want to just get some text and work out the API actions but a lot of sites use horribly obfuscated Javascript -- either because that's what modern web development is (lolz) -- or because its part of their 'security.' That means if you want to write browser-based bots properly -- you ought to use a browser. There are spe…

Just npm install puppeteer.
Post reply on HN