Live data from Hacker News

CLI tools hidden in the Python standard library

til.simonwillison.net

91–100 of 160 posts

Re: CLI tools hidden in the Python standard library

#91
post #22

Speaking of hidden Python tools, I'm a big fan of re.Scanner[0]. It's a regex-based tokenizer[1] in the `re` module, that for reasons is completely missing from any official documentation. You give it a pattern for each token type, and a function to be called on each match, and you get back a list of processed tokens. Importantly, it processes the list in one pass and ensures the matches are contiguous, where a naive…

FWIW there is an example of building a tokenizer/scanner using documented features of the re module here: https://docs.python.org/3.11/library/re.html#writing-a-token... re.Scanner looks more succinct though...!

I find that even more bizarre tbh. Thst seems like the perfect place to document the Scanner class.

Re: CLI tools hidden in the Python standard library

#92
post #10

There is one problem with those: Security. Using modules (even if they are in the standard library) on the command line lets malicious code in the current dir take over your machine: https://twitter.com/marekgibney/status/1598706464583028736

Doing that requires palcement of files right in the directory where the user is likely to run that module.

Seems to be a quite rare vector for exploitation.

Sure, on a multiuser system I might trick some other user into running such a command in /tmp and prepare that directory accordingly, but other vectors seem more esoteric.

Re: CLI tools hidden in the Python standard library

#93
post #41
post #39

Earlier quoted context omitted.

Especially now that they are on a crusade against their own stdlib. They regret including most modules… it seems they regret making python altogether instead of sticking with C? :D

This is a typical programmer, you'll regret code as soon as you turn your back on it. :)

Or before!

Re: CLI tools hidden in the Python standard library

#94
post #38

Earlier quoted context omitted.

There's no magic, only layers. `python -m pdb ` runs pdb with the rest of the arguments. pdb handles the second `-m`. If you have a fancy IDE feature, open a new python file, type "import pdb", use go to definition on pdb to jump to that file in the standard library, and read its main function - it handles -m explicitly :)

Speaking of pdb.. maybe someone knows why pdb has some issues with scope in its REPL that are resolved in VSCode's debugger and PyCharm? Multiline statements are not accepted, nor things like if/for Even list comprehensions and lambda expressions have trouble loading local variables defined via the REPL Are there workarounds? It would reduce the need for using IDEs. People who have experience with Julia and Matlab ar…

Not supporting multi-line statements is just because pdb doesn't bother to parse the statement to work out if it is an incomplete multi-line statement. That could be easily fixed (I have a prototype patch for that using `code.compile_command`).

The scope problems are more fundamental:

The pdb REPL calls[1] the exec builtin as `exec(code, frame.f_globals, frame.f_locals)`, which https://docs.python.org/3/library/functions.html#exec documents as:

"If exec gets two separate objects as globals and locals, the code will be executed as if it were embedded in a class definition."

And https://docs.python.org/3/reference/executionmodel.html#reso... documents that:

"The scope of names defined in a class block is limited to the class block; it does not extend to the code blocks of methods - this includes comprehensions and generator expressions since they are implemented using a function scope."

This is a fundamental limitation of `exec`. You can workaround it by only passing a single namespace dictionary to exec instead of passing separate globals and locals, which is what pdb's interact command does[2], but then it's ambiguous how to map changes to that dictionary back to the separate globals and locals dictionaries (pdb's interact command just discards any changes you make to the namespace). This too could be solved, but requires either brittle ast parsing or probably a PEP to add new functionality to exec. I'll file a bug against Python soon.

[1]: https://github.com/python/cpython/blob/25a64fd28aaaaf2d21fae...

[2]: https://github.com/python/cpython/blob/25a64fd28aaaaf2d21fae...

Re: CLI tools hidden in the Python standard library

#95
post #80
post #63

Earlier quoted context omitted.

> They regret including most modules… it seems they regret making python altogether instead of sticking with C? :D I also find it odd. Python would probably be a little known language without the huge batteries included by default. It's invaluable when you're working in a environment where you can't fully control what is installed, what is the case of most people at work. I believe this crusade endangers the language…

You're reading a ridiculous mischaracterization of reality (~20 deprecated modules [1] out of the ~300 PSL modules [2] is "most"?). Of course you find it odd. [1] https://peps.python.org/pep-0594/ [2] https://docs.python.org/3/library/

It's the trend that is a bit worrying. It's definitely been floated more than once that the stdlib should be fundamentally gutted. It has not become a reality yet, but there are some loud voices advocating to just chuck most modules as a principle into pypi.

Re: CLI tools hidden in the Python standard library

#96
post #10

There is one problem with those: Security. Using modules (even if they are in the standard library) on the command line lets malicious code in the current dir take over your machine: https://twitter.com/marekgibney/status/1598706464583028736

Doing that requires palcement of files right in the directory where the user is likely to run that module. Seems to be a quite rare vector for exploitation. Sure, on a multiuser system I might trick some other user into running such a command in /tmp and prepare that directory accordingly, but other vectors seem more esoteric.

There are thousands of tools (shellscripts, makefiles ...) which execute "python -m":

https://www.gnod.com/search/?engines=af&nw=1&q=python%20-m

Even in Google's own repos. Starting any of those (no matter where they are stored) in a hostile repo would let the code in the repo take over the machine.

Re: CLI tools hidden in the Python standard library

#97
post #95
post #80

Earlier quoted context omitted.

You're reading a ridiculous mischaracterization of reality (~20 deprecated modules [1] out of the ~300 PSL modules [2] is "most"?). Of course you find it odd. [1] https://peps.python.org/pep-0594/ [2] https://docs.python.org/3/library/

It's the trend that is a bit worrying. It's definitely been floated more than once that the stdlib should be fundamentally gutted. It has not become a reality yet, but there are some loud voices advocating to just chuck most modules as a principle into pypi.

"Trend"? Python has dropped quite a few modules over the decades.

"gl", "sgi", "fl", "sunaudiodev", "audioop", "stdwin", "rotor", "poly", "whatsound", "gopherlib" .. the list goes on.

Some of these were dropped with 1.6 (see "Obsolete Modules" at https://www.python.org/download/releases/1.6/ ). Some with 3.0 (see https://docs.python.org/3.10/whatsnew/2.6.html for then-upcoming removals.)

Re: CLI tools hidden in the Python standard library

#99
post #68

I am not someone who complains about "stop piping cats" but using '-e' with grep is a lot quicker and easier to read. This: grep -v 'test/' | grep -v 'tests/' | grep -v idlelib | grep -v turtledemo Becomes: grep -ve 'test/' -e 'tests/' -e idlelib -e turtledemo

To be fair, the advantage of doing separate greps is working iteratively and drilling down on what you want.

Re: CLI tools hidden in the Python standard library

#100
post #73
post #10

There is one problem with those: Security. Using modules (even if they are in the standard library) on the command line lets malicious code in the current dir take over your machine: https://twitter.com/marekgibney/status/1598706464583028736

I find the entire premise of the post to be pretty baffling. > Seth pointed out this is useful if you are on Windows and don't have the gzip utility installed. Okay, so instead of installing gzip (or just using the decompressors that aren't the official gzip utility but that do support the format and already ship with Windows by default[1]), you install Python...? Even if the premise weren't muddy from the start, the…

You can definitely be in situations where you have python but not gzip.
Post reply on HN