Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Boring Python: Code quality

b-list.org

Post reply on HN
Print view View original

21–30 of 232 posts

Re: Boring Python: Code quality

#21

Post by bombolo »

sgtnoodle wrote: post #19
eesmith wrote: post #14

Earlier quoted context omitted.

> All those big changes introduce commits that make git bisect generally slower. Bisection search is log2(n) so doubling the number of commits should only add one more bisection step, yes? > Which might be awful if you also have some C code to recompile at every step of bisecting. That reminds me, I've got to try out ccache ( https://ccache.dev/ ) for my project. My full compile is one minute, but the three files tha…

Perhaps the poster meant that the contents of the commits themselves make bisection slower? By touching a lot of files unnecessarily, incremental build systems have to do more work than otherwise.

Both… and version jumps in formatting tool basically will touch every single file.

Re: Boring Python: Code quality

#22

Post by gjulianm »

anderskaseorg wrote: post #12

If you aren’t happy with Flake8, Pylint, and isort (or maybe if you are!), I recommend checking out Ruff: https://github.com/charliermarsh/ruff It’s literally 100 times faster, with comparable coverage to Flake8 plus dozens of plugins, automatic fixes, and very active development.

Does ruff replace isort? Because I'm really unhappy with it, it doesn't work with tabs and conflicts with yapf all the time.

Re: Boring Python: Code quality

#23

Post by eesmith »

bombolo wrote: post #16
eesmith wrote: post #14

Earlier quoted context omitted.

> All those big changes introduce commits that make git bisect generally slower. Bisection search is log2(n) so doubling the number of commits should only add one more bisection step, yes? > Which might be awful if you also have some C code to recompile at every step of bisecting. That reminds me, I've got to try out ccache ( https://ccache.dev/ ) for my project. My full compile is one minute, but the three files tha…

> Bisection search is log2(n) so doubling the number of commits should only add one more bisection step, yes? And testing 1 extra step could only add a 1 hour build more, yes?

It could, certainly. But

1) you don't have one black commit for every non-black commit, do you? Because the general best practice is to do like kuu suggested and have a specific black version as part of the development environment, with a pre-commit hook to ensure no random formatting gets introduced.

2) assuming 500 commits in your bisection, that's, what, about 9 compilations you'll need to do, so it will take you 9 hours to run. So even with a black commit after every human commit, that yes, 1 hour more, but it's also only 11% longer.

Even with only 10 non-black commits and 10 black commits, your average bisect time will only increase from 3.6 hours to 4.6 hours, or 30% longer.

I'm curious to know what project you are on with 1 hour build times and the regular need for hours-long bisection search, but where there isn't a common Python dev environment with a specific black version. Are you using ccache and/or distributed builds? If not, why isn't there a clear economic justification for improving your build environment? I mean, I assume developers need to build and test before commit, which means each commit is already taking an hour. Isn't that a waste of expensive developer time?

And, I assume it's not the black formatting changes which result in hour-long builds. If they do, could you explain how?

Re: Boring Python: Code quality

#25

Post by eesmith »

sgtnoodle wrote: post #19
eesmith wrote: post #14

Earlier quoted context omitted.

> All those big changes introduce commits that make git bisect generally slower. Bisection search is log2(n) so doubling the number of commits should only add one more bisection step, yes? > Which might be awful if you also have some C code to recompile at every step of bisecting. That reminds me, I've got to try out ccache ( https://ccache.dev/ ) for my project. My full compile is one minute, but the three files tha…

Perhaps the poster meant that the contents of the commits themselves make bisection slower? By touching a lot of files unnecessarily, incremental build systems have to do more work than otherwise.

Could be? I've not heard of that problem, but I don't know everything.

I'm not used to Python code (which is all that black touches) as being notably slow to build, nor am I used to incremental build systems for Python byte compilation.

And I expect in a project with 2 developers which is big enough for things to be slow, then most of the files will be unchanged semantically speaking, only swapping back and forth between two syntactically re-blackened representations, so wouldn't an caching build system be able to cache both forms?

(NB: I said "caching build system" because an incremental build system which expects time linear order, wouldn't be that helpful in bisection, which jumps back-and-forth through the commits.)

Re: Boring Python: Code quality

#26

Post by fnord123 »

bombolo wrote: post #15
kuu wrote: post #9

Earlier quoted context omitted.

> Black formats things differently depending on the version. Then add black as part of your environment with an specific version...

Or wait until a more sensible formatting tool comes along. Reformatting the whole code every version isn't so good. It's also very slow.

Install pre-commit: https://pre-commit.com/

Set black up in the pre-commit with a specific version. When you make a commit it will black the files being committed using the specific version of black. As it's a subset, it's fast. As it's a specific version, it's not going back and forth.

I hope this solves your issues.

Re: Boring Python: Code quality

#27

Post by wodenokoto »

kuu wrote: post #9
bombolo wrote: post #7

> I recommend using two tools together: Black and isort. Black formats things differently depending on the version. So a project with 2 developers, one running arch and one running ubuntu, will get formatted back and forth. isort's completely random… For example the latest version I tried decided to alphabetically sort all the imports, regardless if they are part of standard library or 3rd party. This is a big change…

> Black formats things differently depending on the version. Then add black as part of your environment with an specific version...

That would only make it more likely that two developers would be using two different versions of Black.

The further you get away from the project folder the more likely each developer is to have a different environment.

Re: Boring Python: Code quality

#28

Post by anderskaseorg »

gjulianm wrote: post #22
anderskaseorg wrote: post #12

If you aren’t happy with Flake8, Pylint, and isort (or maybe if you are!), I recommend checking out Ruff: https://github.com/charliermarsh/ruff It’s literally 100 times faster, with comparable coverage to Flake8 plus dozens of plugins, automatic fixes, and very active development.

Does ruff replace isort? Because I'm really unhappy with it, it doesn't work with tabs and conflicts with yapf all the time.

Yes, as of last month. I’m not sure if it works with YAPF; it’s designed to work with Black and doesn’t currently have many of isort’s configuration options. Worth a try!

Re: Boring Python: Code quality

#29

Post by bvrmn »

> Coverage measurements are too easy to “game” — you can get to 100% coverage without meaningfully testing all or even most of your code

Still it's a good low bar for testing. It's easy and rises code quality. I have very good results with coverage driving colleagues to write tests. And on code review we can discuss how to make tests more useful and robust and how to decrease number of mocks, etc.

Re: Boring Python: Code quality

#30

Post by adammarples »

wodenokoto wrote: post #27
kuu wrote: post #9

Earlier quoted context omitted.

> Black formats things differently depending on the version. Then add black as part of your environment with an specific version...

That would only make it more likely that two developers would be using two different versions of Black. The further you get away from the project folder the more likely each developer is to have a different environment.

Just put a versioned black into pre-commit yaml and put that in your source and forget about it
Post reply on HN