Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Python types have an expectations problem

medium.com

Post reply on HN
Print view View original

51–60 of 115 posts

Re: Python types have an expectations problem

#51

Post by saghm »

ehutch79 wrote: post #6

A note; typescript does nothing to ensure types are correct at runtime. Especially in the browser. You need to do runtime checking even if you're using typescript. In Python, yeah, they're called Type _Hints_ for a reason. Don't count on them at runtime here either. Both are dynamic languages, it's hard doing anything meta/schema driven with rigid types. If you really want a hard type system, just move to GO or Rust…

To be fair, TypeScript is a bit different because you can use it to compile to JavaScript rather than merely annotating in JavaScript files, which is crucial because it means that compilation can _fail_ to produce JavaScript. If I recall correctly, you can still have it produce JavaScript even if the type checking fails, but the fact that you can't just directly run the TypeScript makes a difference. With the Python type annotations, you can still just run the code directly without _ever_ compiling, let alone if type checking fails.

> If you really want a hard type system, just move to GO or Rust or C or something with a real type system enforced.

For what it's worth, Rust also has no types at runtime, which is why Rust doesn't have reflection and relies heavily on compile-time magic with macros. The difference is that there's no way to get a binary to run if the code doesn't typecheck. The most underrated feature of a compiler is that it can say "no"; in the case of Rust, the killer feature of the language isn't what you're allowed to do, but what you're _not_ allowed to do, even accidentally.

Re: Python types have an expectations problem

#52

Post by sesgoe »

pphysch wrote: post #46
sesgoe wrote: post #8

From my experience working on an older Python codebase, this issue is definitely a headache. It's extremely difficult to gradually adopt typing in an older Python codebase with almost no typing information because the only real "enforcement" option seems to be a CI pipeline running something like `mypy`. This issue compounds in a painful way. Because 99% of your codebase is starting out untyped, you have a couple of…

I'm interested what the motivation is for retroactively typing an (untyped) legacy codebase. And how far will you go with it? Are you converting bespoke dicts to sensible NamedTuples/dataclasses? Or are you purely adding type hints?

There's a ton of motivation from a DX perspective -- Python type hinting's arguably least valuable feature is the autocompletion improvement that comes with it.

It's nice to be able to hit `.` in your editor and have the options for whatever object you're staring at pop into a list you can pick from. Similarly, for a `TypedDict`, you can hit `["` and just have all the possible key options autocomplete.

The bespoke dicts mostly come from early-days SQL queries akin to

  SELECT * FROM single_table
I wrote a small tool in Rust (yes, I was looking for an excuse to use Rust at work) first to create a giant `TypedDict` file that basically typed all the table rows in our primary database and then a small parser that reads our codebase for these trivial select * from single_table queries and adds `TypedDict` hinting for autocompletion purposes going forward.

So a line like:

  some_result = curs.fetchone()
becomes:

  some_result: SingleTable = curs.fetchone()
Then when you type:

  some_result["
You get a lovely list of auto-completed possible keys instead of having to go hunting through the table to remember what the column was called.

The other reasons we wanted to implement typing were all the main reasons you'd want a typed codebase in the first place. Shift a lot of mistakes to build-time instead of deployed-in-production time.

Re: Python types have an expectations problem

#53

Post by sesgoe »

sco1 wrote: post #27
sesgoe wrote: post #8

From my experience working on an older Python codebase, this issue is definitely a headache. It's extremely difficult to gradually adopt typing in an older Python codebase with almost no typing information because the only real "enforcement" option seems to be a CI pipeline running something like `mypy`. This issue compounds in a painful way. Because 99% of your codebase is starting out untyped, you have a couple of…

> you gradually add more and more files to the `mypy` invocation until you're eventually (hopefully) adding entire subfolders, and then maybe eventually the entire codebase Rather than doing this, which does indeed seem like a headache, it may make more sense to skip import following at the very beginning until your core is typed so you can still enforce typing on the leaf nodes moving forward. > Eventually you get t…

> Rather than doing this, which does indeed seem like a headache, it may make more sense to skip import following at the very beginning until your core is typed so you can still enforce typing on the leaf nodes moving forward.

Yea, this is solid advice and something we did at one point. It's essentially strictly necessary for an older codebase.

> While there are some cases where this is truly the best option, ultimately you get to the point where you just don't allow this, otherwise what's the point of all the effort?

Again, true! We reached fatigue and gave up long before we hit the point where this would have mattered.

> The faster the core can be typed (and typed correctly), the easier it becomes for those who are less passionate. Presumably someone has done the calculus to determine that this effort is worthwhile, so while the team doesn't necessarily all have to reach guru level, they need to be convinced to continue the work. Removing barriers is huge for this, since as you've noticed once it starts being easy to ignore it's really challenging to stop ignoring.

I think this was probably my biggest failure in terms of the success of adding typing. I severely underestimated how long it would take to add typing info to some of the really old pieces of code, and it wasn't reasonable to expect to be able just to sit down and add types without delivering business value for an extended period of time.

My inexperience with `mypy` and the general typing ecosystem in python contributed significantly to the team ultimately reaching fatigue and deciding to give up on it (mostly).

Re: Python types have an expectations problem

#54

Post by maxfurman »

robertlagrant wrote: post #33
dboreham wrote: post #26

Earlier quoted context omitted.

Funny, but for the readers who don't know C -- this isn't true. C has type checking at compile time.

It has something, but not what you'd call modern type checking: #include int main(void) { unsigned int positive_number = -1; printf("%d", positive_number); return 0; } Prints -1.

I must be missing something here. Wouldn't an unsigned int be unable to represent -1 since the sign bit is part of the value?

Re: Python types have an expectations problem

#55

Post by sesgoe »

hauntsaninja wrote: post #30
sesgoe wrote: post #8

From my experience working on an older Python codebase, this issue is definitely a headache. It's extremely difficult to gradually adopt typing in an older Python codebase with almost no typing information because the only real "enforcement" option seems to be a CI pipeline running something like `mypy`. This issue compounds in a painful way. Because 99% of your codebase is starting out untyped, you have a couple of…

Lean heavier on a mypy config file. Make heavy use of per-module configuration, in particular, setting per-module ignore_errors = true for modules you’re not yet ready to type check. See https://mypy.readthedocs.io/en/stable/existing_code.html for some more advice.

Please listen to this advice for anyone giving this a go after reading these comments.

I was pretty green to the Python typing ecosystem when I started implementing it in our large pre-existing codebase, and I did not lean heavily enough on a `mypy` configuration that was module-specific.

It would have saved me a significant headache, and in hindsight, this seems like the main viable option for typing an old codebase effectively. This gets extremely gross when you have 300 or 400+ submodules across your codebase, but start small and work your way from the outside in if you want the best chance of success.

Re: Python types have an expectations problem

#56

Post by lpapez »

magnusmundus wrote: post #37
lpapez wrote: post #18

Earlier quoted context omitted.

I did that for Python, and now I do it for PHP as well. Trivial to set up but ends up being a huge time saver, especially when reviewing junior colleagues code - don't even ping me to review your code until you've managed to convince the static analyzer it will work!

Re: junior colleagues, do you not instead arrive at "why does the static analyzer complain", or worse, an avoidance of proper types that you then have to point out in review?

You definitely do, for the first few times.

But after that they learn to write code which makes both the static analyzer and the reviewer happy, and such code tends to be much more maintainable down the line.

When you don't have strict typing discipline enforced by the CI, you will likely have to enforce it manually anyway because projects in a gradually typed language without simple types enforced tend to become a complete mess IMO.

Re: Python types have an expectations problem

#57

Post by mg »

zokier wrote: post #50
mg wrote: post #16

I don't really like types on the function declaration line. Instead of this: def check_permission(user: User, perm: str, obj: BaseModel | None) -> bool: I find it much nicer to read something this: """ Check if user is allowed to drive a car :user: User # The application's User model :perm: str # Our magic permission string. Tab seperated. :obj : Basemodel | None # Base model of all our objects since 2017 :returns bo…

What do you think of the py27 backcompat type hint syntax using `# type:` comments: https://peps.python.org/pep-0484/#suggested-syntax-for-pytho...

Same. I don't like that the meta stuff inside the function. I prefer to have the function be pure code.

Re: Python types have an expectations problem

#58

Post by michaelt »

sevensor wrote: post #49
ehutch79 wrote: post #6

A note; typescript does nothing to ensure types are correct at runtime. Especially in the browser. You need to do runtime checking even if you're using typescript. In Python, yeah, they're called Type _Hints_ for a reason. Don't count on them at runtime here either. Both are dynamic languages, it's hard doing anything meta/schema driven with rigid types. If you really want a hard type system, just move to GO or Rust…

Python's type hints are great as machine-checkable statements about constraints on the behavior of your code. It's not strictly true that they're unavailable at runtime. > Don't count on them at runtime here either If you're adventurous enough, you can reflect on the type hints and check things yourself at runtime, but you have to understand that the type hints aren't meant for this and they could well blow up in you…

> It's not strictly true that they're unavailable at runtime.

So is it possible for me to enforce type hints at runtime, so my program crashes if a type is ever wrong? How do I turn this checking on?

Re: Python types have an expectations problem

#59

Post by ReflectedImage »

Using static typing in Python is a serious mistake. The point of static typing is to allow an compiler to compile your code for performance reasons and nothing else. People do not normally compile Python code with a compiler.

Static typing significantly increases code length compared to duck typing, significantly increases development times and significantly increases bug count per delivered software feature.

It additionally gives developers the false impression that you can write large projects in Python without splitting the codebase into small micro-services. Hint: You can't, it is always a complete disaster as Python's support for static typing isn't good enough for large projects.

Re: Python types have an expectations problem

#60

Post by empath-nirvana »

ReflectedImage wrote: post #59

Using static typing in Python is a serious mistake. The point of static typing is to allow an compiler to compile your code for performance reasons and nothing else. People do not normally compile Python code with a compiler. Static typing significantly increases code length compared to duck typing, significantly increases development times and significantly increases bug count per delivered software feature. It addi…

> significantly increases bug count per delivered software feature.

Going to need a source for this.

Post reply on HN