Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How to create a Python package in 2022

mathspp.com

Post reply on HN
Print view View original

141–150 of 153 posts

Re: How to create a Python package in 2022

#141

Post by emacs28 »

remram wrote: post #133
emacs28 wrote: post #59

Earlier quoted context omitted.

Thanks, I think that the thing this still lacks compared to `pip install -e` is that the develop dependency is then only visible inside other packages configured with poetry. So it is not useful for example if you want to locally install a package you make to be accessed through cli or any other python script not using poetry.

I'm not sure what you are talking about. It will be installed in the virtual environment just the same. There is no mechanism for it to "only be visible inside other packages configured with poetry".

What I mean is `pip install -e` installs it outside of the virtual environment also.

Re: How to create a Python package in 2022

#142

Post by doliveira »

fnord123 wrote: post #138
doliveira wrote: post #74

Earlier quoted context omitted.

> 1. Because it defeats the purpose of pre-commit. No, it doesn't. Pre-commits are just a DX bonus so you won't have to wait for the CI/CD server. 2. pre-commit install installs the hook by default. Am I missing something? If you run "$ pre-commit install" of course it will install the hooks. Do you mean that poetry install, setup.py themselves are making the magic of installing the hooks themselves? If so, it's inde…

Filling the CI/CD pipeline with tiny commits to fix linting when it should be done before commit or push is why pre-commit exists. It's not BS, clogging the job queue shits in everyone else. Of course I don't know GPs specific context so they may have other details, but generally it's

Yeah, I agree. It's just that they can just run pre-commit manually if the hooks bother them that much

Re: How to create a Python package in 2022

#143

Post by remram »

emacs28 wrote: post #141
remram wrote: post #133

Earlier quoted context omitted.

I'm not sure what you are talking about. It will be installed in the virtual environment just the same. There is no mechanism for it to "only be visible inside other packages configured with poetry".

What I mean is `pip install -e` installs it outside of the virtual environment also.

No, it doesn't.

Re: How to create a Python package in 2022

#144

Post by wyuenho »

fnord123 wrote: post #138
doliveira wrote: post #74

Earlier quoted context omitted.

> 1. Because it defeats the purpose of pre-commit. No, it doesn't. Pre-commits are just a DX bonus so you won't have to wait for the CI/CD server. 2. pre-commit install installs the hook by default. Am I missing something? If you run "$ pre-commit install" of course it will install the hooks. Do you mean that poetry install, setup.py themselves are making the magic of installing the hooks themselves? If so, it's inde…

Filling the CI/CD pipeline with tiny commits to fix linting when it should be done before commit or push is why pre-commit exists. It's not BS, clogging the job queue shits in everyone else. Of course I don't know GPs specific context so they may have other details, but generally it's

Except this problem rarely comes up. Everyone has their favorite way to set up their editors to lint and format automatically using the project's config.

Over my career, I see that every company has some version of:

1. New guy joins a team, wants to push code ASAP

2. He sets up his favorite editor and punt configuring linters and formatters

3. Code pushed, CI fails at linting

4. Some well-meaning coworker or new guy suggests some variations of husky/pre-commit/fancy git hook scripts

5. Team agrees and put that into the repo

6. 6 months to 1 year later the entire team realizes its benefit does not out-weight the cost, and unanimously agree to rip it out unceremoniously.

Re: How to create a Python package in 2022

#145

Post by emacs28 »

remram wrote: post #143
emacs28 wrote: post #141

Earlier quoted context omitted.

What I mean is `pip install -e` installs it outside of the virtual environment also.

No, it doesn't.

There is a standardized API for package tools to expose and invoke editable-install functionality [1] which pip install follows, but Poetry didn't implement this API before (I see it does now though as of this past February [2]).

In other words, this a more thorough explanation of my point from [3]:

> Yes, if you ran poetry install you could get editable mode, but that requires every single end user of your package to install poetry and explicitly invoke a poetry install just for your package. And if someone else's package uses a different package builder, now they need to install and invoke that one. And on and on, and you end up with a six-hundred-line install script because of all the one-off "must install this developer's favorite package manager to use their package" stuff.

[1] https://peps.python.org/pep-0660/

[2] https://github.com/python-poetry/poetry/issues/34#issuecomme...

[3] https://www.reddit.com/r/Python/comments/t3p3ub/comment/hyum...

Re: How to create a Python package in 2022

#146

Post by fnord123 »

wyuenho wrote: post #144
fnord123 wrote: post #138

Earlier quoted context omitted.

Filling the CI/CD pipeline with tiny commits to fix linting when it should be done before commit or push is why pre-commit exists. It's not BS, clogging the job queue shits in everyone else. Of course I don't know GPs specific context so they may have other details, but generally it's

Except this problem rarely comes up. Everyone has their favorite way to set up their editors to lint and format automatically using the project's config. Over my career, I see that every company has some version of: 1. New guy joins a team, wants to push code ASAP 2. He sets up his favorite editor and punt configuring linters and formatters 3. Code pushed, CI fails at linting 4. Some well-meaning coworker or new guy…

Well I have different experiences where no one would ever go back to living without pre-commit, black, autoflake, etc. It moves the fixes to before they get to CI. There's literally no downside.

Maybe you have experience with some weird tool? Or running black and autoflake messed around with emacs and it needs to reload all the buffers?

Re: How to create a Python package in 2022

#147

Post by wyuenho »

fnord123 wrote: post #146
wyuenho wrote: post #144

Earlier quoted context omitted.

Except this problem rarely comes up. Everyone has their favorite way to set up their editors to lint and format automatically using the project's config. Over my career, I see that every company has some version of: 1. New guy joins a team, wants to push code ASAP 2. He sets up his favorite editor and punt configuring linters and formatters 3. Code pushed, CI fails at linting 4. Some well-meaning coworker or new guy…

Well I have different experiences where no one would ever go back to living without pre-commit, black, autoflake, etc. It moves the fixes to before they get to CI. There's literally no downside. Maybe you have experience with some weird tool? Or running black and autoflake messed around with emacs and it needs to reload all the buffers?

I initially had some problems, then I've written an emacs package[1] to fix all of them. And it's due to the course of writing this package made me realize just how bad pre-commit is from its UX, design to its entire premise.

[1]: https://github.com/wyuenho/emacs-python-exec-find

Re: How to create a Python package in 2022

#148

Post by int_19h »

zeotroph wrote: post #43
diekhans wrote: post #23

Lot of good info and saved away! However, it drinks the code coverage cool-aid that started like 30 years ago when code coverage tools emerged. Management types said "high test code coverage == high quality"; lets bean count that!! A great way to achieve high code coverage is to have less than robust code that does not check for crazy error cases that are really hard to reproduce in test cases. Code coverage is a too…

Python is the language with one of the highest 100%-coverage-to-effort ratios. The included unittest.mock framework is making it quite easy to trigger obscure errors and ensure they are handled properly. Combined with thoughtful use of `# pragma: no cover` a 98% code coverage nowadays is an immediate warning that something was rushed. With this and type checking I feel RuntimeErrors much easier to avoid these days. A…

The problem with using mocks extensively for testing is that you then end up mostly testing the mocks. You'll know you've hit that point when you have 100% coverage, but things still break routinely due to interaction (sometimes very indirect) between components.

Re: How to create a Python package in 2022

#149

Post by wyuenho »

areeh wrote: post #139
wyuenho wrote: post #57

I wish people would just forget about pre-commit, this thing is especially useless in a setting where a CI/CD pipeline exists. It's not that hard to write a simple Makefile or shellscript to run linters on push. Pre-commit is one of the most annoying tools that have come into existence in recent years that everyone seems to be cargo-culting. It doesn't play well with editors since in order to find the actual binary p…

It's convenient to run the lint step faster/sooner than at CI/CD time. Depending on your setup, the separate linter deps handled by pre-commit can be more convenient than hassle both locally and in your CI/CD pipeline (re the makefile script you mention). Having done it both ways several times I lean pre-commit for now

You've already run them at edit time.

Re: How to create a Python package in 2022

#150

Post by wyuenho »

doliveira wrote: post #100
wyuenho wrote: post #78

Earlier quoted context omitted.

> One thing that's really annoying these days are CI/CD that can't be replicated locally pre-commit just runs your linter, formatter and tests. Surely you can fully replicate this step locally. Just run make lint both locally and on CI. Anything else that's hard to replicate has to do with the distributed systems that you are probably working on because these systems are all probably proprietary stuff that live on th…

My point is that using these tools at least makes people put these scripts within the repo instead of just in the CI server.

There's no "instead of" to speak of. Either you run the same linting step both locally and on CI, or just on CI, you can't skip linting on CI. Every properly set up CI pipeline has some combination of Makefile/shell script/Dockerfile that runs exactly the same way locally and on CI, which the script checked into the repo itself. If your CI scripts don't exist in your repo, you are doing it wrong.
Post reply on HN