Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

CLI tools hidden in the Python standard library

til.simonwillison.net

Post reply on HN
Print view View original

21–30 of 160 posts

Re: CLI tools hidden in the Python standard library

#21

Post by TeMPOraL »

chmaynard wrote: post #17
polyrand wrote: post #13

Python 3.12 will include a SQLite CLI/REPL in the standard library too[0][1]. This is useful because most operating systems have sqlite3 and python3, but are missing the SQLite CLI. [0]: https://github.com/python/cpython/blob/3fb7c608e5764559a718c... [1]: https://docs.python.org/3.12/library/sqlite3.html#command-li...

> This is useful because most operating systems have sqlite3 and python3, but are missing the SQLite CLI. Not sure what you mean -- sqlite3 is the SQLite CLI.

They likely mean libsqlite3, i.e. the .DLL/.so.

Re: CLI tools hidden in the Python standard library

#22

Post by BoppreH »

Speaking of hidden Python tools, I'm a big fan of re.Scanner[0]. It's a regex-based tokenizer[1] in the `re` module, that for reasons is completely missing from any official documentation.

You give it a pattern for each token type, and a function to be called on each match, and you get back a list of processed tokens.

Importantly, it processes the list in one pass and ensures the matches are contiguous, where a naive `re.findall` with capture groups will ignore unmatched characters. You also get a reference to the running scanner, so you can record the location of the match for reporting errors.

    import re
    scanner = re.Scanner([
      (r"[0-9]+",       lambda scanner, token:("INTEGER", int(token))),
      (r"[a-z_]+",      lambda scanner, token:("IDENTIFIER", token)),
      (r"[,.]+",        lambda scanner, token:("PUNCTUATION", token)),
      (r"\s+", None), # None == skip token.
    ])

    results, remainder = scanner.scan("45 pigeons, 23 cows, 11 spiders.")
    assert not remainder
    print(results)

    [('INTEGER', 45),
     ('IDENTIFIER', 'pigeons'),
     ('PUNCTUATION', ','),
     ('INTEGER', 23),
     ('IDENTIFIER', 'cows'),
     ('PUNCTUATION', ','),
     ('INTEGER', 11),
     ('IDENTIFIER', 'spiders'),
     ('PUNCTUATION', '.')]
[0]: https://stackoverflow.com/a/693818/252218

[1]: https://en.wikipedia.org/wiki/Lexical_analysis#Tokenization

Re: CLI tools hidden in the Python standard library

#23

Post by Timon3 »

BoppreH wrote: post #22

Speaking of hidden Python tools, I'm a big fan of re.Scanner[0]. It's a regex-based tokenizer[1] in the `re` module, that for reasons is completely missing from any official documentation. You give it a pattern for each token type, and a function to be called on each match, and you get back a list of processed tokens. Importantly, it processes the list in one pass and ensures the matches are contiguous, where a naive…

It seems like the discussion of whether to document it died in April of 2003: https://mail.python.org/pipermail/python-dev/2003-April/0350...

Bummer, it's a cool feature, but I don't feel safe relying on undocumented features.

Re: CLI tools hidden in the Python standard library

#25

Post by 9dev »

> Pretty-print JSON: > echo '{"foo": "bar", "baz": [1, 2, 3]}' | python -m json.tool

This is even more fun on MacOS if you combine it with the pbpaste/pbcopy utils:

  alias json_pretty="pbpaste | python -m json.tool | pbcopy"
That command will pretty-print any JSON in your clipboard, and write it back to the clipboard, so you can paste it somewhere else formatted!

Re: CLI tools hidden in the Python standard library

#26

Post by masklinn »

Shame gzip has one but zlib does not, that would be a very useful addition: some software create raw zlib streams on disk (e.g. git) and there’s no standard decompressor, you need to either prepend a fake gzip header, go through openssl, qpdf‘s zlib-flate, or pigz -z.

Re: CLI tools hidden in the Python standard library

#27

Post by BiteCode_dev »

sharikous wrote: post #18
BiteCode_dev wrote: post #11

Earlier quoted context omitted.

It also uses the file as a module, not a script, which means suddenly relative imports works the root dir and the cwd are the same, and it is added to sys.path. This prevents a ton of import problems, albeit for the price of more verbose typing, especially since you don't have completion on dotted path. It is my favorite way of running my projects. Unfortunalty it means you can't use "-m pdb", and that's a big loss.

You can use pdb Just python -m pdb -m module

Damn, 15 years of python and I learn you can use -m twice. I've never even tried, didn't occur to me it would be supported.

EDIT: it support other options as well, like -c. That deserves an alias:

    debug_module() {
        if python -c "import ipdb" &>/dev/null; then
            python -m ipdb -c c -m "$@"
        else
            python -m pdb -c c -m  "$@"
        fi
    }

Re: CLI tools hidden in the Python standard library

#28

Post by matsemann »

BoppreH wrote: post #22

Speaking of hidden Python tools, I'm a big fan of re.Scanner[0]. It's a regex-based tokenizer[1] in the `re` module, that for reasons is completely missing from any official documentation. You give it a pattern for each token type, and a function to be called on each match, and you get back a list of processed tokens. Importantly, it processes the list in one pass and ensures the matches are contiguous, where a naive…

> completely missing from any official documentation

To be fair, most things are missing from the official documentation. When I learned kotlin, I read through their official docs, and knew about most language features in a day. When I learned python, I constantly got surprised by things I hadn't seen come up in the docs. For instance decorators was (still is?) not mentioned at all in the official tutorial.

Re: CLI tools hidden in the Python standard library

#29

Post by BoppreH »

matsemann wrote: post #28
BoppreH wrote: post #22

Speaking of hidden Python tools, I'm a big fan of re.Scanner[0]. It's a regex-based tokenizer[1] in the `re` module, that for reasons is completely missing from any official documentation. You give it a pattern for each token type, and a function to be called on each match, and you get back a list of processed tokens. Importantly, it processes the list in one pass and ensures the matches are contiguous, where a naive…

> completely missing from any official documentation To be fair, most things are missing from the official documentation. When I learned kotlin, I read through their official docs, and knew about most language features in a day. When I learned python, I constantly got surprised by things I hadn't seen come up in the docs. For instance decorators was (still is?) not mentioned at all in the official tutorial.

Decorators seem to be documented now:

https://docs.python.org/3/glossary.html#term-decorator

https://docs.python.org/3/reference/compound_stmts.html#func...

Re: CLI tools hidden in the Python standard library

#30

Post by abhishekjha »

BiteCode_dev wrote: post #27
sharikous wrote: post #18

Earlier quoted context omitted.

You can use pdb Just python -m pdb -m module

Damn, 15 years of python and I learn you can use -m twice. I've never even tried, didn't occur to me it would be supported. EDIT: it support other options as well, like -c. That deserves an alias: debug_module() { if python -c "import ipdb" &>/dev/null; then python -m ipdb -c c -m "$@" else python -m pdb -c c -m "$@" fi }

Like whats the general usage though?

python -m http.server is the most I have done.

Post reply on HN