Ask HN: How do you deal with large Python code bases? • Hacker News Board

Re: Ask HN: How do you deal with large Python code bases?

#11

Post by MichaelRazum » Thu, Jun 01, 2023, 9:17 AM UTC

StefanWestfal wrote: post #10

MichaelRazum wrote: post #4

What exactly are your problem's? It is hard to give advice without knowing. To be honest I don't think it makes things easier if you switch to Go, since less verbose.

It gives you a lot of freedom and doesn't force many rules like other non-dynamic languages do. This makes it great for trying out ideas and working with data and machine learning. However, when you're writing more complex programs, you need to be more careful. That's why I feel like I encounter more bugs during runtime compared to other languages. I mentioned Go because it makes you stick to a smaller number of ways…

Hard to tell but that sounds a bit like a test problem. If you have high test coverage (which you must do in a language like python) most bugs should only result through complex interaction.

Re: Ask HN: How do you deal with large Python code bases?

#12

Post by BerislavLopac » Tue, Jun 06, 2023, 7:40 PM UTC

jstx1 wrote: post #3

How are you finding mypy for your project? I find that I waste a lot of time telling mypy to ignore some library that it doesn't understand, much more than I'm saving by catching type errors (which is never). People add it to projects because it seems like a best practice and the proper thing to do, but I'm kind of unconviced that it's making things better.

> telling mypy to ignore some library that it doesn't understand

Such as?

Re: Ask HN: How do you deal with large Python code bases?

#13

Post by fazkan » Tue, Jun 06, 2023, 7:44 PM UTC

pg_1234 wrote: post #6

Regression tests. Static typing still doesn't help for the more insidious issues - errors of value.

I used to use dataclasses for value assertions, but have switched to use https://www.attrs.org/en/stable/. Its like the one tool you will need for 90% of the cases.

Re: Ask HN: How do you deal with large Python code bases?

#14

Post by fazkan » Tue, Jun 06, 2023, 7:48 PM UTC

hayst4ck wrote: post #7

The larger your codebase gets the more bazel becomes a requirement. Bazel is really not negotiable for large python code bases. The more bazel is put off, the more pain you will endure before you eventually are forced to use bazel. You will be forced to use bazel or a system like it because eventually your good devs will not tolerate your codebase and leave without it. https://bazel.build/ Other than bazel you will h…

I thought bazel was only needed for C++ builds? haven't seen it used in Python projects, is the utility of bazel for the C++ binaries that scientific libraries like numpy, pandas use?

Re: Ask HN: How do you deal with large Python code bases?

#15

Post by dstromberg » Tue, Jun 06, 2023, 10:38 PM UTC

Python _is_ typed. Python is Dynamically Typed. The word "Dynamically" does not suddenly erase the word "Typed". Even CPython is Dynamically Typed.

You presumably mean Statically, Manifestly Typed.

Assembly Language and Forth are Untyped. The closest thing they have to types are bytes and cells.

For examples of a language that is Statically but not Manifestly Typed, have a look at Shedskin (which is a dialect of Python) or Haskell (which is a fascinating Functional, Lazy language with a regrettably poor compiler).

I continue to see mypy, or something like it, as the best way of taking a medium sized Python project into the world of large projects. For smaller projects, ruff or pylint are sufficient, but for large projects, mypy and similar are the way to go.