How to make a Python package in 2021 • Hacker News Board

Re: How to make a Python package in 2021

#121

Post by BerislavLopac » Thu, Apr 08, 2021, 8:43 AM UTC

zomglings wrote: post #8

The number of extra tools used in this article boggles my mind. Are you writing a simple library? Create a setup.py. Copy paste an existing setup.py and modify it to suit your purposes. Now you have a working, pip installable python package. Want to publish to PyPI? Use twine. It's standard and it's simple. You don't need complicated tooling for simple projects.

Using setup.py does not mean "not using extra tools". It depends on setuptools, which is an "extra tools" just like flit (used in the article) or any other tool. In fact, with using only setuptools one will need a whole additional set of tools to manage things like:

    * virtual environments (granted, venv is now part of the stdlib, but it's still an "extra tool")
    * publishing to PyPI or another index (twine)
    * dependency management (both for development and actual dependencies)

Plus the tools that are needed anyway, to manage common development actions like:

    * unit tests (pytests)
    * static typing (mypy)
    * linting (flake8, pylint)
    * styling (black)

The article is correct in using pyproject.toml, which has become the standard way to specify build mechanism for your package [0]. Even setuptools supports it in the latest versions [1], meaning that setup.py is becoming obsolete, or at least unnecessary.

Finally, tools like Poetry [2] offer a whole set of functionalities in one place (dependency management, virtual environments, publishing), which means that they need fewer "extra tools" than just setuptools.

[0] https://www.python.org/dev/peps/pep-0518/

[1] https://setuptools.readthedocs.io/en/latest/build_meta.html

[2] https://python-poetry.org/

Re: How to make a Python package in 2021

#122

Post by afarrell » Thu, Apr 08, 2021, 8:45 AM UTC

hardwaresofton wrote: post #109

darthrupert wrote: post #81

Earlier quoted context omitted.

I actually think one should do the exact opposite to what you're suggesting. I've experienced modern package management through Cargo and anything below that level now seems like returning to stone age. In the python ecosystem, poorly defined packages is a wide problem. You never know what you get in the next version upgrade. So my suggestion: burn all the "simple" python packaging tools in a big fire and move eveyth…

Bit of a side note but remember cargo learned a lot from generations of package managers, across languages. It essentially represents some of the best that package managers have to offer. You only get that kind of result by starting from scratch every few years (as in, rust started from scratch when the Node/NPM ecosystem had been around for years to steal ideas from, Haskell had been around for years to steal langua…

Migrating python2 -> python3 is hard because it requires rewriting code and because you can't really run python3 code if you depend upon python2.

If poetry create consume non-poetry packages and create packages which other package management systems can consume? If so, then:

1. Projects can move more effectively independently.

2. Projects can re-package as a single task rather than a massive rewrite effort.

Re: How to make a Python package in 2021

#123

Post by BerislavLopac » Thu, Apr 08, 2021, 8:47 AM UTC

guggle wrote: post #54

neolog wrote: post #3

Just use Poetry [1]. It's popular and works well. [1] https://python-poetry.org/

That's a 72MB download, and yet another way to fragment the ecosystem. Not something I'd get just to make a package when a default Python setup has recommended tools and everything I need to make and/or install packages.

> a default Python setup has recommended tools and everything I need to make and/or install packages

No it doesn't. Neither setuptools nor pip are part of the standard library. Yes, they are installed by default in many cases, but they are still "extra tools".

Re: How to make a Python package in 2021

#124

Post by nxpnsv » Thu, Apr 08, 2021, 8:53 AM UTC

anewhnaccount2 wrote: post #71

icy wrote: post #42

Honestly, I don't even bother "packaging" Python tools anymore. Just put it in all in a git repo, and pip can install using pip install git+https://myg.it/repo.git

But now packages on PyPI can't depend on it.

But very often, that's ok...

Re: How to make a Python package in 2021

#125

Post by BerislavLopac » Thu, Apr 08, 2021, 8:56 AM UTC

74d-fe6-2c6 wrote: post #108

Can somebody confirm that this is reasonable and state of the art? Or is it just a "look what I can do" type blog post?

I would say it is quite up to date. First, it is using pyproject.toml which is now the standard way to define build requirements for Python packages. Second, its collection of additional (linting etc) tools is pretty solid; there are potential alternatives in few cases (e.g. I would personally use Poetry rather than flit, and wouldn't use make for development scripts) but that is pretty much it.

Re: How to make a Python package in 2021

#126

Post by toyg » Thu, Apr 08, 2021, 8:57 AM UTC

neolog wrote: post #87

xapata wrote: post #63

Earlier quoted context omitted.

On the whole, Guido's career as a BDFL was astoundingly effective. Maybe he made the right call. It'd have been a terrible idea to alienate the science community just when data science was taking off as a field.

I don't know how much the original author had to do with Python's success in the last 20 years. The success of Python in data science is because of NumPy/Scipy/Pandas. Things like packaging that needed leadership never got any.

> The success of Python in data science is because of NumPy/Scipy/Pandas.

And NumPy/Scipy/Pandas exist (and are successful) because...? /s

Re: How to make a Python package in 2021

#127

Post by BerislavLopac » Thu, Apr 08, 2021, 8:58 AM UTC

zomglings wrote: post #15

neolog wrote: post #11

Earlier quoted context omitted.

How do you handle version pinning? hash checking? CI? testing on multiple platforms? multiple python versions? deployment? credential management? package data? version bumps? Sure, experts know how to do all these things because they spent many days learning them, but I'd rather outsource to a tool.

Iteratively. You don't need to solve all those problems at once. Version pinning can be done in setup.py using the same syntax you would see in a requirements.txt file. You should be very conservative when pinning versions in a library, though. You can lean on your ci tool (eg. Github actions) to handle testing, hash checking, credential management, etc. But I recommend all of this start as a bunch of locally runnabl…

> You should be very conservative when pinning versions in a library, though

No; you should be very conservative when pinning versions in an application, not in a library. Check this article for the explanation: https://caremad.io/posts/2013/07/setup-vs-requirement/

Re: How to make a Python package in 2021

#128

Post by edenhyacinth » Thu, Apr 08, 2021, 8:59 AM UTC

74d-fe6-2c6 wrote: post #108

Can somebody confirm that this is reasonable and state of the art? Or is it just a "look what I can do" type blog post?

It's more the latter, if you read the rest of the thread you'll get a feel for the issue - there's no broad consensus on what Python should do for package management.

Flit is not an unreasonable pick, but it's not a silver bullet.

Re: How to make a Python package in 2021

#129

Post by nxpnsv » Thu, Apr 08, 2021, 9:00 AM UTC

simonw wrote: post #65

I have a cookiecutter template for this at https://github.com/simonw/python-lib (also click-app and datasette-plugin) It sets up GitHub actions for publishing the package to PyPI when you create a release on GitHub - which I find way to be a really productive way of working.

cookiecutter is great. I recently moved from click to typer, which I so far really have enjoyed. I probably should make a cc template for that one day...

Re: How to make a Python package in 2021

#130

Post by cixinlui » Thu, Apr 08, 2021, 9:06 AM UTC

zomglings wrote: post #8

The number of extra tools used in this article boggles my mind. Are you writing a simple library? Create a setup.py. Copy paste an existing setup.py and modify it to suit your purposes. Now you have a working, pip installable python package. Want to publish to PyPI? Use twine. It's standard and it's simple. You don't need complicated tooling for simple projects.

This is bad advice. Do not create a setup.py for a new package. (Keeping setup.py for an old package can be okay.)

The author is correct that you want a tool such as flit or poetry which will work with pyproject.toml. Setting up a basic package will be no harder than using setuptools, and it is much more future-proof. You won't have to copy-paste some other crufty setup config either.

It is fair that you don't need all the external tools in this tutorial. In particular, using make is very silly since you can configure different linting and testing workflows directly in pyproject.toml, rather than pull in a whole other system which only works decently on *nix. Poetry also removes the need for tox.