Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

YouTube-dl has an interpreter for a subset of JavaScript in 870 lines of Python

twitter.com

Post reply on HN
Print view View original

61–70 of 166 posts

Re: YouTube-dl has an interpreter for a subset of JavaScript in 870 lines of Python

#61

Post by sebzim4500 »

jraph wrote: post #55
esprehn wrote: post #42

This isn't really JS, it's a purpose built evaluator that's only for evaluating a particular script on YouTube, assuming a huge list of things are true about how YouTube JS is written. Ex. Its got a hard coded list of methods for String, and it doesn't respect prototypes. It only supports creating Date instances, and won't work if you override the global Date. It parses with regexes and implements all operators with…

And as a user of youtube-dl, I'm quite happy about this. This probably allows a very safe, restricted "subset" of JS. Way better than using a full JS engine. 900 lines is still small and manageable.

I'm trying to get the thread model here. Is the concern that Youtube will inject JS into the payload which tries to break out of the youtuble-dl js sandbox using some zero day in whatever js engine they would use instead?

Re: YouTube-dl has an interpreter for a subset of JavaScript in 870 lines of Python

#62

Post by chess_buster »

Test0129 wrote: post #53
baobabKoodaa wrote: post #52

Earlier quoted context omitted.

There's a huge difference between an interpreter for "JavaScript" and an interpreter for a "subset of JavaScript".

Making a pedantic argument on what constitutes an interpreter is silly. The title is bad. It is an interpreter. I'll continue to eat downvotes on this because of the pedantry of HN.

I evaluated it with my Pedantic Interpreter which only results in the `pedantic` token.

Re: YouTube-dl has an interpreter for a subset of JavaScript in 870 lines of Python

#63

Post by [deleted] »

Test0129 wrote: post #48
esprehn wrote: post #42

This isn't really JS, it's a purpose built evaluator that's only for evaluating a particular script on YouTube, assuming a huge list of things are true about how YouTube JS is written. Ex. Its got a hard coded list of methods for String, and it doesn't respect prototypes. It only supports creating Date instances, and won't work if you override the global Date. It parses with regexes and implements all operators with…

This really isn't fair. Just because it doesn't faithfully implement whatever standard Javascript is on doesn't mean it isn't an interpreter. All an interpreter is is something that executes a script directly rather than requiring compilation. It is a defacto interpreter for a subset of javascript. Nothing more, nothing less. The title could be more clear, however.

[deleted]

Re: YouTube-dl has an interpreter for a subset of JavaScript in 870 lines of Python

#64

Post by mjevans »

jraph wrote: post #55
esprehn wrote: post #42

This isn't really JS, it's a purpose built evaluator that's only for evaluating a particular script on YouTube, assuming a huge list of things are true about how YouTube JS is written. Ex. Its got a hard coded list of methods for String, and it doesn't respect prototypes. It only supports creating Date instances, and won't work if you override the global Date. It parses with regexes and implements all operators with…

And as a user of youtube-dl, I'm quite happy about this. This probably allows a very safe, restricted "subset" of JS. Way better than using a full JS engine. 900 lines is still small and manageable.

yt-dlp sometimes doesn't know how to evaluate the javascript / emcascript and will call out to an optional dependency, a real javascript interpreter, if installed.

Re: YouTube-dl has an interpreter for a subset of JavaScript in 870 lines of Python

#65

Post by kevingadd »

sebzim4500 wrote: post #61
jraph wrote: post #55

Earlier quoted context omitted.

And as a user of youtube-dl, I'm quite happy about this. This probably allows a very safe, restricted "subset" of JS. Way better than using a full JS engine. 900 lines is still small and manageable.

I'm trying to get the thread model here. Is the concern that Youtube will inject JS into the payload which tries to break out of the youtuble-dl js sandbox using some zero day in whatever js engine they would use instead?

Embedding a whole js engine and then interopping with it from python would be non trivial. Good luck fixing any bugs or corner cases you hit that way. The V8 and spidermonkey embedding apis are both c++ (iirc) and non trivial to use correctly.

Having full control like this +simple code is probably lower risk and more maintainable, even if there's the challenge of expanding feature set if scripts change.

The alternative would be a console js shell, but those are very different from browsers so that poses it's own challenges.

Re: YouTube-dl has an interpreter for a subset of JavaScript in 870 lines of Python

#66

Post by blast »

Test0129 wrote: post #48
esprehn wrote: post #42

This isn't really JS, it's a purpose built evaluator that's only for evaluating a particular script on YouTube, assuming a huge list of things are true about how YouTube JS is written. Ex. Its got a hard coded list of methods for String, and it doesn't respect prototypes. It only supports creating Date instances, and won't work if you override the global Date. It parses with regexes and implements all operators with…

This really isn't fair. Just because it doesn't faithfully implement whatever standard Javascript is on doesn't mean it isn't an interpreter. All an interpreter is is something that executes a script directly rather than requiring compilation. It is a defacto interpreter for a subset of javascript. Nothing more, nothing less. The title could be more clear, however.

esprehn didn't say it isn't an interpreter. They're saying it is an interpreter and what it's interpreting isn't (all of) JS. That's also what you're saying, so you're agreeing with esprehn.

Edit: You misunderstood baobabKoodaa in the same way. Nobody is arguing about what constitutes an interpreter, except you. The question is what language is being interpreted.

Before accusing someone of pedantry, it would first be good not to completely misread them.

Re: YouTube-dl has an interpreter for a subset of JavaScript in 870 lines of Python

#67

Post by jraph »

sebzim4500 wrote: post #61
jraph wrote: post #55

Earlier quoted context omitted.

And as a user of youtube-dl, I'm quite happy about this. This probably allows a very safe, restricted "subset" of JS. Way better than using a full JS engine. 900 lines is still small and manageable.

I'm trying to get the thread model here. Is the concern that Youtube will inject JS into the payload which tries to break out of the youtuble-dl js sandbox using some zero day in whatever js engine they would use instead?

Let's say they end up using Node. Node has a quite complete standard library that lets you access files and everything.

Now if they do it right and only embed some bare JS interpreter, it's still way harder to audit than these < 900 lines, for which it is quite easy to convince oneself that the interpreted script cannot do much.

Re: YouTube-dl has an interpreter for a subset of JavaScript in 870 lines of Python

#68

Post by baobabKoodaa »

Test0129 wrote: post #53
baobabKoodaa wrote: post #52

Earlier quoted context omitted.

There's a huge difference between an interpreter for "JavaScript" and an interpreter for a "subset of JavaScript".

Making a pedantic argument on what constitutes an interpreter is silly. The title is bad. It is an interpreter. I'll continue to eat downvotes on this because of the pedantry of HN.

> Making a pedantic argument on what constitutes an interpreter is silly. The title is bad. It is an interpreter.

It's not a pedantic argument. Based on the title I thought that somebody wrote something akin to V8 in 800 lines of Python. After reading the comments I realized those 800 lines just interpret a particular JavaScript function written by Youtube. Those things are different. Pointing out the fact that they are different is not pedantry. The title is misleading and the comments pointing that out are helpful.

Re: YouTube-dl has an interpreter for a subset of JavaScript in 870 lines of Python

#69

Post by kristopolous »

To understand why, I have a far simpler tool that focuses on a subset of sites (adult content video aggregators)

https://github.com/kristopolous/tube-get

It too deals with this problem but does so in a way that'd be easy to maliciously sabotage

Look right about here https://github.com/kristopolous/tube-get/blob/master/tube-ge...

As to why this program exists, this was originally written between about 2010-2015 or so technically predates the yt-* ecosystem.

The tool still works fine and it's not a strict subset of yt-dlp or YouTube-dl because being a different approach, although it's overall site coverage is smaller, I've had it be a "second try" system when yt-* fails and it comes up with success maybe about half the time

Re: YouTube-dl has an interpreter for a subset of JavaScript in 870 lines of Python

#70

Post by blast »

esprehn wrote: post #42

This isn't really JS, it's a purpose built evaluator that's only for evaluating a particular script on YouTube, assuming a huge list of things are true about how YouTube JS is written. Ex. Its got a hard coded list of methods for String, and it doesn't respect prototypes. It only supports creating Date instances, and won't work if you override the global Date. It parses with regexes and implements all operators with…

I suppose this means it would be easy for YouTube to fuck with youtube-dl simply by throwing in more features of JS?
Post reply on HN