How to make a Python package in 2021 • Hacker News Board

Re: How to make a Python package in 2021

#171

Post by oefrha » Thu, Apr 08, 2021, 12:59 PM UTC

chriswarbo wrote: post #150

fctorial wrote: post #140

Earlier quoted context omitted.

For all the hate nodejs gets, it solved the software packaging problem. That is the sole reason npm ecosystem is so big. I have to say that it probably isn't a fare comparison because python is much older than nodejs. Python package management might very well have been state of the art in 1995.

My impression was that npm does little more than fetch code from the Web and stick it in a 'node_modules' directory. I've even seen npm used for things that aren't even JS, just bunch of files. This approach ends up with multiple, potentially-incompatible versions of the same package in a project. True that's less of a problem in JS, since it's interpreted (deferring imports to runtime) and un(i)typed (no need to che…

> My impression was that npm does little more than fetch code from the Web and stick it in a 'node_modules' directory.

Yes. There's hardly even a standard directory structure, let alone a standard way to convert source code to published code. Every slightly non-trivial repo basically has an ad hoc build system of its own. Ever tried to fix a bug in a package, and realized that using git://github.com/user/repo#branch doesn't work, because npm only downloads the source code, which bears no resemblance to the built products? I fixed two bugs in two third party packages within the past week, had to deal with this twice. Ran into the Node 12+ and Gulp 3.x incompatibility issue twice in the past month (with totally modern, actively developed packages), too.

npm has more sophisticated dependency resolution and locking than pip, sure. Python packaging is more consistent in basically every other regard.

Re: How to make a Python package in 2021

#172

Post by cixinlui » Thu, Apr 08, 2021, 1:00 PM UTC

1337shadow wrote: post #157

cixinlui wrote: post #130

Earlier quoted context omitted.

This is bad advice. Do not create a setup.py for a new package. (Keeping setup.py for an old package can be okay.) The author is correct that you want a tool such as flit or poetry which will work with pyproject.toml. Setting up a basic package will be no harder than using setuptools, and it is much more future-proof. You won't have to copy-paste some other crufty setup config either. It is fair that you don't need a…

And why exactly do you think setup.py is deprecated?

Because of PEP 518.

Re: How to make a Python package in 2021

#173

Post by wdroz » Thu, Apr 08, 2021, 1:08 PM UTC

When you are doing machine learning, conda is widely used. Why? Because you can install non-python things like cudatoolkit or ffmpeg (you can even install python, so you are sure that everybody are using the same version of python)

Python is fantastic at gluing specialized tools/libraries, but a lot of these require non-python dependencies (most are written in more performant languages). IMO, this is a big differences when comparing with Cargo for Rust because most of the dependencies in Rust are written in Rust.

The state of packaging in Python is kinda meh, the official documentation here [0] suggests to create 3 additional files in order to create a package:

  - pyproject.toml
  - setup.cfg
  - setup.py  # optional, needed to make editable pip installs work

if you add conda, you may need 2 additional files:

  - meta.yaml  # to build you conda package
  - environment.yml

With this much boilerplate, I understand why people are creating tools like flit.

[0] -- https://packaging.python.org/tutorials/packaging-projects/

Re: How to make a Python package in 2021

#174

Post by losvedir » Thu, Apr 08, 2021, 1:15 PM UTC

tcbasche wrote: post #24

neolog wrote: post #3

Just use Poetry [1]. It's popular and works well. [1] https://python-poetry.org/

How does it compare to pipenv?

I just moved a project from pipenv to poetry at work. My biggest issue with pipenv is that you can't selectively upgrade dependencies. Trying to `pipenv update xyz` basically blows away your lockfile and updates everything. There's a command line flag to be more selective but it doesn't work. I found an open GitHub issue about it that's years old.

Poetry by contrast works pretty much like any modern dependency system you'd be familiar with from another language like cargo, npm, or hex.

Re: How to make a Python package in 2021

#175

Post by patagonia » Thu, Apr 08, 2021, 1:19 PM UTC

vonwoodson wrote: post #166

zomglings wrote: post #8

The number of extra tools used in this article boggles my mind. Are you writing a simple library? Create a setup.py. Copy paste an existing setup.py and modify it to suit your purposes. Now you have a working, pip installable python package. Want to publish to PyPI? Use twine. It's standard and it's simple. You don't need complicated tooling for simple projects.

Comments on this reply seem to have forgotten two key things about python. 1) It’s batteries included. The standard library is a core feature, and it is extensive in order to reduce the number 3rd party library dependencies. 2) “There should be one— and preferably only one —obvious way to do it. Although that way may not be obvious at first unless you're Dutch.” The Pythonic way is to use the standard library, setup.…

Unpopular opinion: Python is bait and switch. Naive, aspiring programmers are told to learn Python because it’s the perfect beginner language, only to quickly hit a brick wall of complication they are unable to foresee (because they’re completely new to all this) and ill-equipped navigate.

Like it or not Python is now “enterprise grade”. The sooner it either grows up and figures out packages, ci/cd, binaries etc, or it looses its reputation at a great beginner and general programming language to something like Julia or Crystal, the better.

Re: How to make a Python package in 2021

#176

Post by cixinlui » Thu, Apr 08, 2021, 1:21 PM UTC

vonwoodson wrote: post #166

zomglings wrote: post #8

The number of extra tools used in this article boggles my mind. Are you writing a simple library? Create a setup.py. Copy paste an existing setup.py and modify it to suit your purposes. Now you have a working, pip installable python package. Want to publish to PyPI? Use twine. It's standard and it's simple. You don't need complicated tooling for simple projects.

Comments on this reply seem to have forgotten two key things about python. 1) It’s batteries included. The standard library is a core feature, and it is extensive in order to reduce the number 3rd party library dependencies. 2) “There should be one— and preferably only one —obvious way to do it. Although that way may not be obvious at first unless you're Dutch.” The Pythonic way is to use the standard library, setup.…

The thing is that using setuptools is neither standard nor Pythonic. It isn't part of the standard library. It's a way of doing things that is broken, and has been specifically called out by the Python developer community as something that people should stop doing.

Re: How to make a Python package in 2021

#177

Post by guggle » Thu, Apr 08, 2021, 1:55 PM UTC

cixinlui wrote: post #136

guggle wrote: post #54

Earlier quoted context omitted.

That's a 72MB download, and yet another way to fragment the ecosystem. Not something I'd get just to make a package when a default Python setup has recommended tools and everything I need to make and/or install packages.

72MB? On a development machine? Are you on dialup? Poetry doesn't fragment the ecosystem. Unlike setuptools it uses pyproject.toml, which can be read by other tools, and is the correct way of storing package configuration. A package built using Poetry is installable without Poetry in the exact same way as one built using setuptools.

Last time I tried poetry it was so broken that it was not even usable. I may try again later.

Re: How to make a Python package in 2021

#178

Post by amarshall » Thu, Apr 08, 2021, 2:14 PM UTC

1337shadow wrote: post #102

amarshall wrote: post #97

Earlier quoted context omitted.

Probably because setupmeta seems to only supports setup.py, which this guide doesn’t use (and isn’t generally recommended for most use cases).

I maintain over 50 public packages with setup.py and most of them have setupmeta, it's great and I would recommend it for all use cases.

Sure, and that’s fine. My statement was based off https://packaging.python.org/tutorials/packaging-projects/

> dynamic metadata [(setup.py)] should be used only as an escape hatch when absolutely necessary.

Re: How to make a Python package in 2021

#179

Post by regularfry » Thu, Apr 08, 2021, 2:32 PM UTC

brundolf wrote: post #45

zomglings wrote: post #30

Earlier quoted context omitted.

I sympathize. It is unfortunate that the python community never settled around a tool like leiningen for clojure or cargo for rust or npm for node. What we saw with npm was the entire community iterating towards a feature set and everyone reaping the benefits automatically with npm updates. package-lock.json is a good example of this.

Worth noting is that cargo and npm weren't "settled around"; they were developed and presented, from the beginning, alongside the relevant compiler and runtime. There was never a question; the batteries were included. Leiningen is the weird one where people did actually settle fairly well around an unofficial solution in the absence of an official one. I think the norm with languages that forego official tooling is c…

Rubygems, and then Bundler, followed the same pattern. Neither was batteries-included, both were unofficial community efforts. Bundler directly influenced cargo.

Re: How to make a Python package in 2021

#180

Post by asdf123wtf » Thu, Apr 08, 2021, 2:33 PM UTC

neolog wrote: post #11

zomglings wrote: post #8

The number of extra tools used in this article boggles my mind. Are you writing a simple library? Create a setup.py. Copy paste an existing setup.py and modify it to suit your purposes. Now you have a working, pip installable python package. Want to publish to PyPI? Use twine. It's standard and it's simple. You don't need complicated tooling for simple projects.

How do you handle version pinning? hash checking? CI? testing on multiple platforms? multiple python versions? deployment? credential management? package data? version bumps? Sure, experts know how to do all these things because they spent many days learning them, but I'd rather outsource to a tool.

I generally reach for pip-tools, when I need to pin versions in a requirements for a deployable app, like an api.

It's by far the simplest option I've found.

If your project is a library, just use setup.py and express your deps abstractly (min, max or version range). Don't pin to specific versions at all, if you can help it.