How to create a Python package in 2022 • Hacker News Board

Re: How to create a Python package in 2022

#21

Post by quantumduck » Thu, Jul 28, 2022, 6:42 AM UTC

plonk wrote: post #18

lmeyerov wrote: post #15

We've been fine-ish with classic setup.py/setup.cfg + gha for publishing to pypi. But as we do OSS data science (gpu graph ai + viz), where conda is typical nowadays... ... Have to admit: We recently ended up contracting conda packaging out because it was nowhere near clear enough to make sense for our core team to untangle. Would love to see a similar tutorial on a github flow packaging & publishing to conda. Still…

We do a lot of deep learning and image processing and pip works much better for us. PyTorch makes wheels that contain all reauired DLLs on all systems. Maybe conda isn’t needed anymore.

Conda let's you install a specific cuda version directly in a virtual environment with one click though. It's really useful when you have to switch between multiple PyTorch versions and convinient in general imo.

Re: How to create a Python package in 2022

#22

Post by notemaker » Thu, Jul 28, 2022, 6:45 AM UTC

plonk wrote: post #18

lmeyerov wrote: post #15

We've been fine-ish with classic setup.py/setup.cfg + gha for publishing to pypi. But as we do OSS data science (gpu graph ai + viz), where conda is typical nowadays... ... Have to admit: We recently ended up contracting conda packaging out because it was nowhere near clear enough to make sense for our core team to untangle. Would love to see a similar tutorial on a github flow packaging & publishing to conda. Still…

We do a lot of deep learning and image processing and pip works much better for us. PyTorch makes wheels that contain all reauired DLLs on all systems. Maybe conda isn’t needed anymore.

I think of conda as jupyter notebooks. It's ok to experiment with and get something running quickly, but it's a bad idea to use for production code.

Personally I never use either of these tools.

Re: How to create a Python package in 2022

#23

Post by diekhans » Thu, Jul 28, 2022, 6:53 AM UTC

Lot of good info and saved away!

However, it drinks the code coverage cool-aid that started like 30 years ago when code coverage tools emerged.

Management types said "high test code coverage == high quality"; lets bean count that!!

A great way to achieve high code coverage is to have less than robust code that does not check for crazy error cases that are really hard to reproduce in test cases.

Code coverage is a tool to help engineers write good tests. One takes the time to look at the results and improve the test. It is a poor investment to be obsessed with code cover on paths where the cost to test them greatly exceeds the value.

10% coverage and 100% are both alarm bells. Don't assume naive, easy to produce metrics are the same as quality code.

Otherwise, and excellent article.

Re: How to create a Python package in 2022

#24

Post by zx14 » Thu, Jul 28, 2022, 7:05 AM UTC

uranusjr wrote: post #19

zx14 wrote: post #16

Sigh I don't see why I need to use a 3rd party tool for what should be a very straightforward process in Python out of the box. In fact, I think these days it actually is straightforward, of course once you work out what you need to do... Python is a mess.

Just fyi, both npm and yarn are “third party” tools. Cargo is closer to the Rust core, and you can find tools in a similar position in Python as well (e.g. setuptools, hatch). Packaging tools being managed by a different group than people working on the “core language” is actually the norm, since those are very different topics and only very few brilliant people care about both of them at the same time. Python people…

Instead of being insecure about criticisms to the Python ecosystem and calling my disappointment "hatred" I'd rather we focused on solutions.

Just because this mess happens in some other languages doesn't mean it's the right thing. Having a very fragmented community is not a good thing for a beginner. Also, npm is far more of a de facto choice than poetry, which is still better than the state python finds itself in.

Re: How to create a Python package in 2022

#25

Post by RojerGS » Thu, Jul 28, 2022, 7:10 AM UTC

woodruffw wrote: post #6

This is really nicely written; kudos to the author for compiling a great deal of information in a readable format. If I can be forgiven one nitpick: Poetry does not use a PEP 518-style[1] build configuration by default, which means that its use of `pyproject.toml` is slightly out of pace with the rest of the Python packaging ecosystem. That isn't to say that it isn't excellent, because it is! But you the standards ha…

Your nitpick is forgiven! Thanks a lot for this information, I was not aware of this...

However, I took a look at PEP 518 and failed to understand what was wrong with Poetry's default configuration. Can you help me out?

Re: How to create a Python package in 2022

#26

Post by RojerGS » Thu, Jul 28, 2022, 7:10 AM UTC

captn3m0 wrote: post #11

PyPi is adding support for GitHub OIDC for publishing packages soon, so there will be no need to generate API keys - you can just grant your GitHub Actions permissions to publish to PyPi. https://github.com/pypi/warehouse/issues/10619

Oh, this would be neat! Looking forward to it!

Re: How to create a Python package in 2022

#27

Post by RojerGS » Thu, Jul 28, 2022, 7:12 AM UTC

lmeyerov wrote: post #15

We've been fine-ish with classic setup.py/setup.cfg + gha for publishing to pypi. But as we do OSS data science (gpu graph ai + viz), where conda is typical nowadays... ... Have to admit: We recently ended up contracting conda packaging out because it was nowhere near clear enough to make sense for our core team to untangle. Would love to see a similar tutorial on a github flow packaging & publishing to conda. Still…

Oh, I see... I personally barely use conga and I have no idea how that is done. I don't think I'll write any blog article like that any time soon :( Maybe you could do it!

Re: How to create a Python package in 2022

#28

Post by dannyboland » Thu, Jul 28, 2022, 7:12 AM UTC

This is almost exactly how I set up python projects; it’s reassuring to see it set out in one place.

I started using tox-poetry-installer[1] to make tox pick up pinned versions from the lock file and reuse the private package index credentials from poetry.

[1] https://github.com/enpaul/tox-poetry-installer

Re: How to create a Python package in 2022

#29

Post by RojerGS » Thu, Jul 28, 2022, 7:12 AM UTC

tony-allan wrote: post #2

"How do you create a Python package? How do you set up automated testing and code coverage? How do you publish the package? That's what this article teaches you." — delivered as promised!

Original author here: don't make promises you won't keep, right? :P

Re: How to create a Python package in 2022

#30

Post by RojerGS » Thu, Jul 28, 2022, 7:13 AM UTC

davnn wrote: post #7

I guess it's a great exercise to set up a repository yourself in this way, but once you have experience with the technologies involved, it's much easier to just use a cookiecutter template [1] to set up your package. Another aspect to consider is that there are often different tools to achieve the same goal, thus, it makes sense to experiment until you've found your perfect package setup. [1] https://github.com/searc…

Excellent points! I have seen several cookiecutter templates, but like you said, those aren't very useful when you are at the very start and everything looks weird and new.