Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How to create a Python package in 2022

mathspp.com

Post reply on HN
Print view View original

101–110 of 153 posts

Re: How to create a Python package in 2022

#101

Post by woodruffw »

milliams wrote: post #49
woodruffw wrote: post #6

This is really nicely written; kudos to the author for compiling a great deal of information in a readable format. If I can be forgiven one nitpick: Poetry does not use a PEP 518-style[1] build configuration by default, which means that its use of `pyproject.toml` is slightly out of pace with the rest of the Python packaging ecosystem. That isn't to say that it isn't excellent, because it is! But you the standards ha…

I believe that Poetry does conform to PEP 518 (i.e. it specifies `[build-system]requires/build-backend`), but not to the `dependencies` part of PEP 621 [1]. There are plans for this in the future though [2]. Though I would defer to your expertise if I'm mistaken. [1] https://peps.python.org/pep-0621/ [2] https://github.com/python-poetry/roadmap/issues/3

Yes, this was a mistake on my part! I meant PEP 621.

Re: How to create a Python package in 2022

#102

Post by woodruffw »

pid-1 wrote: post #42
woodruffw wrote: post #6

This is really nicely written; kudos to the author for compiling a great deal of information in a readable format. If I can be forgiven one nitpick: Poetry does not use a PEP 518-style[1] build configuration by default, which means that its use of `pyproject.toml` is slightly out of pace with the rest of the Python packaging ecosystem. That isn't to say that it isn't excellent, because it is! But you the standards ha…

> By way of example, here's a project that's completely PEP 517 and PEP 518 compatible without needing a setup.py or setup.cfg[2]. Everything goes through pyproject.toml. Using pyproject.toml with pip / flit still has many rough edges such as pip being unable to install deps locally for development or not generating lock files. Poetry is way more mature IMO.

Maybe I’m misunderstanding what you mean, but installing dependencies locally for development (meaning development extras) and generating lock files (via pep freeze) both work for me.

Re: How to create a Python package in 2022

#103

Post by fluidcruft »

wyuenho wrote: post #57

I wish people would just forget about pre-commit, this thing is especially useless in a setting where a CI/CD pipeline exists. It's not that hard to write a simple Makefile or shellscript to run linters on push. Pre-commit is one of the most annoying tools that have come into existence in recent years that everyone seems to be cargo-culting. It doesn't play well with editors since in order to find the actual binary p…

Do you have any tutorials for setting up CI/CD? My impression was that's all stuff that runs in the cloud but if it's something I can use on my own personal projects I'd play with it. Frankly a lot of these things become unintelligible. I've used pre-commit for things like black and autopep8 and that's all pretty understandable to me. The CI/CD things I've read all seem like everyone already understands some giant Rube Goldberg contraption that they're strapping on things for some reason that probably matters to giant dev teams.

Re: How to create a Python package in 2022

#104

Post by cjalmeida »

matheusmoreira wrote: post #95
sigmonsays wrote: post #90

Earlier quoted context omitted.

the irony here is python packaging has sucked forever, and this is just another example of it. "Do more with less" has never entered the average python developers mind. you'd think herd mentality might help it but it only creates more packaging solutions. Now days, I've stopped using python outside of tiny scripts and I will never touch it for a large project.

The problem with Python is it's too successful. It has too many useful libraries. I don't really like it very much but it's really hard to ignore it. There is exactly one thing I miss from Python packaging tools: developer mode. I can factor out parts of an application into a library and develop both at the same time by installing the library in editable mode and pointing to the library's local directory. This is som…

Julia had a dev mode as well

https://pkgdocs.julialang.org/v1/managing-packages/#developi...

It’s packaging tool is top notch. Maybe because it’s developed as a core part of the language.

Re: How to create a Python package in 2022

#105

Post by iggy_knights »

Bullfight2Cond wrote: post #88

Poetry uses non-standard dependency specification formats. PDM is like Poetry but faster/more standards compliant. https://pdm.fming.dev/

Good thing about finally converging on some sort of standard means tools become more interoperable:

Another good tool (which was endorsed by the PyPA) is Hatch - https://hatch.pypa.io/latest/environment/

I currently use PDM because it supports conda virtual environments for isolation, but am keeping an eye on Hatch.

Re: How to create a Python package in 2022

#106

Post by slhck »

udioron wrote: post #86
slhck wrote: post #80

Earlier quoted context omitted.

This is really nice. The only thing I'm missing here is a simple way to bump versions. Any ideas on how to do that? For Node, it's quite simple and even built into npm. Also the version is only part of the package.json file. For Python you probably have your version somewhere in __init__.py, and I always end up writing ugly bash scripts that modify multiple places with sed.

Try commitizen.

I know commitizen and it doesn't do that. It enforces particular forms of commit messages, but it does not bump versions in Python projects.

Re: How to create a Python package in 2022

#107

Post by natpalmer1776 »

fluidcruft wrote: post #103
wyuenho wrote: post #57

I wish people would just forget about pre-commit, this thing is especially useless in a setting where a CI/CD pipeline exists. It's not that hard to write a simple Makefile or shellscript to run linters on push. Pre-commit is one of the most annoying tools that have come into existence in recent years that everyone seems to be cargo-culting. It doesn't play well with editors since in order to find the actual binary p…

Do you have any tutorials for setting up CI/CD? My impression was that's all stuff that runs in the cloud but if it's something I can use on my own personal projects I'd play with it. Frankly a lot of these things become unintelligible. I've used pre-commit for things like black and autopep8 and that's all pretty understandable to me. The CI/CD things I've read all seem like everyone already understands some giant Ru…

Good example of CI/CD that's not a rube goldberg machine per se:

https://github.blog/2022-02-02-build-ci-cd-pipeline-github-a...

Re: How to create a Python package in 2022

#108

Post by iggy_knights »

bvrmn wrote: post #63
dannyboland wrote: post #41

Earlier quoted context omitted.

Yeah, I adopted the “src” dir layout after reading this post some time back: https://blog.ionelmc.ro/2014/05/25/python-packaging/#the-str... It describes some of the outdated motivations for the other layouts commonly seen with python, as well as the many benefits of the “src” layout.

Post doesn't make sense. It clearly states that you only be sure that package is working by actually installing it into clean virtualenv and testing it there. `src` or any other layout doesn't matter.

The difference is if you're running tests from your root project directory, the package is importable regardless of whether or not it is installed, as python picks up packages in the current working directory by name. src/ prevents this.

Re: How to create a Python package in 2022

#109

Post by bvrmn »

iggy_knights wrote: post #108
bvrmn wrote: post #63

Earlier quoted context omitted.

Post doesn't make sense. It clearly states that you only be sure that package is working by actually installing it into clean virtualenv and testing it there. `src` or any other layout doesn't matter.

The difference is if you're running tests from your root project directory, the package is importable regardless of whether or not it is installed, as python picks up packages in the current working directory by name. src/ prevents this.

But it doesn't prevent package configuration errors and allows to bdist broken package. What's the point then?

Re: How to create a Python package in 2022

#110

Post by js2 »

wyuenho wrote: post #57

I wish people would just forget about pre-commit, this thing is especially useless in a setting where a CI/CD pipeline exists. It's not that hard to write a simple Makefile or shellscript to run linters on push. Pre-commit is one of the most annoying tools that have come into existence in recent years that everyone seems to be cargo-culting. It doesn't play well with editors since in order to find the actual binary p…

Pre-commit can also be configured to use the project's dependencies, which is what I do with my repos when there is overlap. You don't have to use its built-in integrations. Indeed, I find its integrations most useful for tools that aren't specific to my project: things like the white-space checks, yaml checks, etc.

You can also set things up the other way around, having the CI/CD system install and and run pre-commit's checks as a build step. Pre-commit provides a nice framework, I find, for running these checks.

The advantage of pre-commit is that it catches mistakes before they are committed. Most CI/CD systems are setup to only validate the tip of the branch (i.e. the last commit in the PR), not all the commits along the way. Yes, you can configure a CI/CD system to test each commit, but it's usually swimming upstream to do so. And yes, you can squash all of a PR's commits into a single commit, but there are good reasons NOT to do that. So assuming you have multiple commits in a change, it's nice to know they have likely all been validated in a project using pre-commmit.

I'll make an appeal to authority here:

I've been a professional developer for decades. I've worked with a variety of VCS's and build systems. I've written plenty of Makefiles. Nonetheless, I still find pre-commit useful. I use it even in combination with a Makefile sometimes. You'd be horrified, I guess, to know some of my Makefiles have a rule which runs `pre-commit run --all`.

As an example, earlier this week I setup a new repo which build packages an AWS Lambda written in javascript, and deploys it using the AWS "sam" CLI via a CI/CD system. So the deployed code is Javascript. The repo contains shell-scripts to assist with deployment. And there are multiple yaml files. There's a yaml file to configure the CI/CD system, and there's the CloudFormation template files.

Here's what I configured pre-commit to do:

1. Check for whitespace nits.

2. Check for syntax errors in the yaml files.

3. Validate the CloudFormation templates.

https://aws.amazon.com/blogs/infrastructure-and-automation/u...

4. Run shellcheck against the shell scripts.

5. Run eslint and prettifier against the javascript.

6. Run "npm test" as a local step.

Normally I'd leave (6) out because in most projects its too time-consuming, but in this project the tests run quickly enough that I just made it a pre-commit check. The "npm test" step runs jest, which is installed as a dev dependency in the project's package.json. The other tools are all installed by pre-commit itself.

> I really don't see the point of any pre-commit hooks unless you are the one guy that doesn't use a modern CI/CD platform.

I've tried to make an argument for why to use pre-commit above. Nevertheless, you don't have to install pre-commit's hooks in your checkout. They are ideally there to save you time having to correct mistakes after the fact. It sounds like there's an impedance mismatch between your personal workflow and how you've seen pre-commit set up. Perhaps by resolving that mismatch by adjusting the pre-commit configuration, you can enjoy pre-commits benefits w/o experiencing the issues you've run into.

Post reply on HN