Live data from Hacker News

YouTube-dl has an interpreter for a subset of JavaScript in 870 lines of Python

twitter.com

121–130 of 166 posts

Re: YouTube-dl has an interpreter for a subset of JavaScript in 870 lines of Python

#121

Earlier quoted context omitted.

I like the Twitter linking since it's almost like the OP is giving credit to where they found the information from.

Agreed. If you only know this from someone else's observation, you should link the observation.

That is against HN guidelines: "Please submit the original source. If a post reports on something found on another site, submit the latter." - https://news.ycombinator.com/newsguidelines.html

Re: YouTube-dl has an interpreter for a subset of JavaScript in 870 lines of Python

#122
post #44
post #6

Earlier quoted context omitted.

They need to run a JavaScript function to download YouTube videos at normal speeds. Edit: it's also required to download music, otherwise it will just fail Source: - https://github.com/ytdl-org/youtube-dl/issues/29326#issuecom... - https://github.com/ytdl-org/youtube-dl/blob/d619dd712f63aab1... - https://github.com/ytdl-org/youtube-dl/commit/cf001636600430...

Wow: Overview of the control flow (already known): The Youtube API provides you with n - your video access token If their new changes apply to your client (they do for "web") then it is expected your client will modify n based on internal logic. This logic is inside player...base.js n is modified by a cryptic function Modified n is sent back to server as proof that we're an official client. If you send n unmodified,…

There is a reason Google is able to serve the amount of video bandwidth, and also a reason why there are no worthwhile youtube clones. Some amount of scrape protection is absolutely essential.

Re: YouTube-dl has an interpreter for a subset of JavaScript in 870 lines of Python

#123
post #19

Nowadays "javascript" refers to the scriptable, grotesquely and absurdely complex and massive web engines, aka google financed blink and geeko, then apple financed webkit, that with their SDK. The currently obfuscated javascript media players will try to break yt-dlp by leveraging the complexity and size of those scripted web engines. They will make them out of reach to small teamns or individuals and it is even "bet…

> quickjs from M. Bellard and friends

Is the M key next to the F key on your particular keyboard by chance? Because I've always called him "Fabrice."

https://en.wikipedia.org/wiki/Fabrice_Bellard

Re: YouTube-dl has an interpreter for a subset of JavaScript in 870 lines of Python

#124
post #72
post #42

This isn't really JS, it's a purpose built evaluator that's only for evaluating a particular script on YouTube, assuming a huge list of things are true about how YouTube JS is written. Ex. Its got a hard coded list of methods for String, and it doesn't respect prototypes. It only supports creating Date instances, and won't work if you override the global Date. It parses with regexes and implements all operators with…

Ok, we've changed this title to shrink the scope of the interpreter. Submitted title was "YouTube-dl has a JavaScript interpreter written in 870 lines of Python".

Hence why HN better than Twitter.

The amount of high engagement just plain wrong tweets there are is just sad.

Re: YouTube-dl has an interpreter for a subset of JavaScript in 870 lines of Python

#125

To understand why, I have a far simpler tool that focuses on a subset of sites (adult content video aggregators) https://github.com/kristopolous/tube-get It too deals with this problem but does so in a way that'd be easy to maliciously sabotage Look right about here https://github.com/kristopolous/tube-get/blob/master/tube-ge... As to why this program exists, this was originally written between about 2010-2015 or so…

Would you mind switching to subprocess with shell=False? os.popen is obsolete and insecure because it passes the command through the shell.

PS: I found it quite easy to contribute to yt-dlp and the reviewers are ultra-helpful and kind, you might want to migrate all of your extractors there.

Re: YouTube-dl has an interpreter for a subset of JavaScript in 870 lines of Python

#126
post #125

To understand why, I have a far simpler tool that focuses on a subset of sites (adult content video aggregators) https://github.com/kristopolous/tube-get It too deals with this problem but does so in a way that'd be easy to maliciously sabotage Look right about here https://github.com/kristopolous/tube-get/blob/master/tube-ge... As to why this program exists, this was originally written between about 2010-2015 or so…

Would you mind switching to subprocess with shell=False? os.popen is obsolete and insecure because it passes the command through the shell. PS: I found it quite easy to contribute to yt-dlp and the reviewers are ultra-helpful and kind, you might want to migrate all of your extractors there.

1. It's ancient code but sure

2. They're fundamentally not compatible approaches. This is worthless to them

Re: YouTube-dl has an interpreter for a subset of JavaScript in 870 lines of Python

#127
post #26

Earlier quoted context omitted.

It will only grow - as new scripts will need to be interpreted, new features will be added.

I would be horrified if this grew much further. It's perfectly fine for its current scope, but the architecture would not scale at all to a full interpreter without essentially starting from scratch.

How much does it need to "scale"? It just has to be fast and correct enough for the a CLI to work.

Re: YouTube-dl has an interpreter for a subset of JavaScript in 870 lines of Python

#128
post #122
post #44

Earlier quoted context omitted.

Wow: Overview of the control flow (already known): The Youtube API provides you with n - your video access token If their new changes apply to your client (they do for "web") then it is expected your client will modify n based on internal logic. This logic is inside player...base.js n is modified by a cryptic function Modified n is sent back to server as proof that we're an official client. If you send n unmodified,…

There is a reason Google is able to serve the amount of video bandwidth, and also a reason why there are no worthwhile youtube clones. Some amount of scrape protection is absolutely essential.

Seems like they ultimately failed, youtube-dl is available freely as a pip package, anyone with scraping intent would have been able to use it.

Re: YouTube-dl has an interpreter for a subset of JavaScript in 870 lines of Python

#129
post #17
post #5

Earlier quoted context omitted.

You need to run some obscured JS to get decent download speeds from Youtube. Something along the lines of PoW.

It’s not like proof of work at all. It’s just a challenge and response; youtube includes a random number in the webpage for each video, and expects to see a request parameter with a particular value calculated from that random number when you request the video. If you don’t do the arithmetic it throttles you to 50kb/s. Since the calculation of the response is done in JS, and they occasionally change the formula, some…

YouTube PM: We need to stop youtube-dl.

Engineers: make half arsed attempt.

Re: YouTube-dl has an interpreter for a subset of JavaScript in 870 lines of Python

#130
Greenspun's Tenth Rule:

> Any sufficiently complicated C or Fortran program contains an ad hoc, informally-specified, bug-ridden, slow implementation of half of Common Lisp. [1]

And here we have a complicated Python program with a partial JS implementation in it.

[1] https://en.wikipedia.org/wiki/Greenspun's_tenth_rule

Post reply on HN