Live data from Hacker News

YouTube-dl has an interpreter for a subset of JavaScript in 870 lines of Python

twitter.com

101–110 of 166 posts

Re: YouTube-dl has an interpreter for a subset of JavaScript in 870 lines of Python

#101
post #42

This isn't really JS, it's a purpose built evaluator that's only for evaluating a particular script on YouTube, assuming a huge list of things are true about how YouTube JS is written. Ex. Its got a hard coded list of methods for String, and it doesn't respect prototypes. It only supports creating Date instances, and won't work if you override the global Date. It parses with regexes and implements all operators with…

[deleted]

Re: YouTube-dl has an interpreter for a subset of JavaScript in 870 lines of Python

#102
post #92

Earlier quoted context omitted.

I'm trying to get the thread model here. Is the concern that Youtube will inject JS into the payload which tries to break out of the youtuble-dl js sandbox using some zero day in whatever js engine they would use instead?

Google attempting zero days on client computers would be something. It's not totally without precedent (Sony CD rootkits - https://en.wikipedia.org/wiki/Sony_BMG_copy_protection_rootk... ) but would still be major news.

While they likely wouldn't do a zero-day, their JS files, particularly for automated captchas, do push the boundaries of whatever JS engine they're executed inside. See https://github.com/neuroradiology/InsideReCaptcha#the-analys... and note that this analysis is 8 years old. While there's minimal risk if you're either using a full-fledged modern JS engine or a limited-subset interpreter like the OP, an older or non-optimized spec-compliant JS engine might hit pathological performance cases and result in you DOSing yourself.

Re: YouTube-dl has an interpreter for a subset of JavaScript in 870 lines of Python

#104

Earlier quoted context omitted.

I'm trying to get the thread model here. Is the concern that Youtube will inject JS into the payload which tries to break out of the youtuble-dl js sandbox using some zero day in whatever js engine they would use instead?

Embedding a whole js engine and then interopping with it from python would be non trivial. Good luck fixing any bugs or corner cases you hit that way. The V8 and spidermonkey embedding apis are both c++ (iirc) and non trivial to use correctly. Having full control like this +simple code is probably lower risk and more maintainable, even if there's the challenge of expanding feature set if scripts change. The alternati…

> Embedding a whole js engine and then interopping with it from python would be non trivial.

Cue libv8-node+mini_racer from which PyMiniRacer was born. It is non-trivial but not as hard as one might think.

The most painful part is the libv8 build system and Google-centric tooling (depot tools!), which makes it an absolute PITA for libv8 consumers that are not Google/Chrome.

This is why the libv8 gem was atrocious to keep up to date and to build for several platforms, and why libv8-node was born, because the node build system and source distribution are actually sane (props to their relentless work on which we piggyback on)

Disclaimer: worked at Sqreen, now maintainer of libv8-node and collaborator of mini_racer

https://github.com/sqreen/PyMiniRacer

https://github.com/rubyjs/mini_racer

https://github.com/rubyjs/libv8-node

Re: YouTube-dl has an interpreter for a subset of JavaScript in 870 lines of Python

#106

Can we stop the trend of linking to tweets that just contain another link to the content? what's the point? Wouldn't this be 10x better if it was a link directly to the github?

I was thinking the same thing; link to the file on Github, with the same title text as is there now, and it saves me an extra click. And any time I don't have to visit Twitter, I consider that a win.

Re: YouTube-dl has an interpreter for a subset of JavaScript in 870 lines of Python

#107
post #26

Earlier quoted context omitted.

It will only grow - as new scripts will need to be interpreted, new features will be added.

I would be horrified if this grew much further. It's perfectly fine for its current scope, but the architecture would not scale at all to a full interpreter without essentially starting from scratch.

Yeah, at some point you have to question if it's worth spending time maintaining a quirky, error-prone, ever-growing mini-JS interpreter, or just adding a dependency on v8 or node or something. And then you don't have to worry about supporting new scripts, as they'll just always work.

Re: YouTube-dl has an interpreter for a subset of JavaScript in 870 lines of Python

#108
post #106

Can we stop the trend of linking to tweets that just contain another link to the content? what's the point? Wouldn't this be 10x better if it was a link directly to the github?

I was thinking the same thing; link to the file on Github, with the same title text as is there now, and it saves me an extra click. And any time I don't have to visit Twitter, I consider that a win.

No post body was provided.

Re: YouTube-dl has an interpreter for a subset of JavaScript in 870 lines of Python

#109

Earlier quoted context omitted.

Chrome and Safari both have open source JS engines…

That's beside the point. Open-source is not useful to the smaller players if it is too complex to comprehend and constantly churned.

That's not the case, though. There are even python modules that let you evaluate JS code in v8 (Chrome's JS interpreter). It'd be pretty trivial for youtube-dl to make use of that if the author thought it was worth doing.

Re: YouTube-dl has an interpreter for a subset of JavaScript in 870 lines of Python

#110
post #107

Earlier quoted context omitted.

I would be horrified if this grew much further. It's perfectly fine for its current scope, but the architecture would not scale at all to a full interpreter without essentially starting from scratch.

Yeah, at some point you have to question if it's worth spending time maintaining a quirky, error-prone, ever-growing mini-JS interpreter, or just adding a dependency on v8 or node or something. And then you don't have to worry about supporting new scripts, as they'll just always work.

If you were going to use a C library, the most logical is QuickJS since it has Python bindings, is small, perfectly fast enough for the kind of needs yt-dl has, and it has excellent coverage of the standard and passes conformance tests.

That said I think a decent Python-native JS interpreter isn't that bad of an idea, it definitely needs a separate project and a more sophisticated architecture but it's an attainable goal.

Post reply on HN