Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Python types have an expectations problem

medium.com

Post reply on HN
Print view View original

21–30 of 115 posts

Re: Python types have an expectations problem

#21

Post by mkesper »

sesgoe wrote: post #8

From my experience working on an older Python codebase, this issue is definitely a headache. It's extremely difficult to gradually adopt typing in an older Python codebase with almost no typing information because the only real "enforcement" option seems to be a CI pipeline running something like `mypy`. This issue compounds in a painful way. Because 99% of your codebase is starting out untyped, you have a couple of…

You could use mypy in pre-commit so code is always checked. Sad there's no per-file switch to tell the interpreter 'this has to be checked'.

Re: Python types have an expectations problem

#22

Post by Malcolmlisk »

mg wrote: post #16

I don't really like types on the function declaration line. Instead of this: def check_permission(user: User, perm: str, obj: BaseModel | None) -> bool: I find it much nicer to read something this: """ Check if user is allowed to drive a car :user: User # The application's User model :perm: str # Our magic permission string. Tab seperated. :obj : Basemodel | None # Base model of all our objects since 2017 :returns bo…

You can do that as a docstring inside the function. There are even some basic rules like...

def extract_coordiates( unreseted_idx_from_df, json_data: dict, notna_idxs: list, na_idxs: list ) -> list:

    """
    Extracting coordinates from the json_data['data'] and returning the 4
    positions of those as an array

    Parameters
    ----------
    json_data: json
        Data extracted from tabula.read_pdf and using output_format='json'
    notna_idxs: list
        List of id's from the dataframe that ha no na values.
    na_idxs: list
        List of id's from the dataframe with na

    Returns
    -------
    List of tuple coordinates:
        [(1, 2), (2, 2), (4, 3), (1, 4)]
    """
Thing is, that you may want to get fast information in the linter with this type hinting. If you need to read the documentation on big functions over and over, that means that's not clear at all, and having basic type hinting while typing the atributes is going to be clearer.

You are right, that some docstrings are necesary. But that does not mean that is the best practice. The best practice is use both, type hinting and docstrings.

Re: Python types have an expectations problem

#23

Post by sesgoe »

mkesper wrote: post #21
sesgoe wrote: post #8

From my experience working on an older Python codebase, this issue is definitely a headache. It's extremely difficult to gradually adopt typing in an older Python codebase with almost no typing information because the only real "enforcement" option seems to be a CI pipeline running something like `mypy`. This issue compounds in a painful way. Because 99% of your codebase is starting out untyped, you have a couple of…

You could use mypy in pre-commit so code is always checked. Sad there's no per-file switch to tell the interpreter 'this has to be checked'.

Yeah, this is something we tried!

And yea, from an IDE perspective, it seems like maybe a sensible default would be to run `mypy ${CURRENT_FILE}` on save or something -- I've tried this manually myself, and it's decent, but it's not good enough to use constantly. I don't remember specific issues since I haven't done this in a long time.

Pre-commits are painful (on purpose!), but preventing people from shipping seemed like the wrong call. So we ended up dropping them.

Re: Python types have an expectations problem

#24

Post by misnome »

tyingq wrote: post #2

There are some other warts also, like hard-to-grok errors if you're using typing syntax that's not yet supported in your version of python. Like: somevar: dict[int, int] = {0: 0, 1: 1} Produces "TypeError: 'type' object is not subscriptable"

Unless you both a) Don't evaluate them at runtime, and b) import annotations from __future__, in which case it is generally safe to write them like this (and AFAIK all of the usual type-checking tools read them fine).

Re: Python types have an expectations problem

#25

Post by ehutch79 »

thadt wrote: post #12
ehutch79 wrote: post #6

A note; typescript does nothing to ensure types are correct at runtime. Especially in the browser. You need to do runtime checking even if you're using typescript. In Python, yeah, they're called Type _Hints_ for a reason. Don't count on them at runtime here either. Both are dynamic languages, it's hard doing anything meta/schema driven with rigid types. If you really want a hard type system, just move to GO or Rust…

C's type checker is the MMU - when you mess up types, it give you a segfault. If you're lucky.

Fair.

Re: Python types have an expectations problem

#26

Post by dboreham »

thadt wrote: post #12
ehutch79 wrote: post #6

A note; typescript does nothing to ensure types are correct at runtime. Especially in the browser. You need to do runtime checking even if you're using typescript. In Python, yeah, they're called Type _Hints_ for a reason. Don't count on them at runtime here either. Both are dynamic languages, it's hard doing anything meta/schema driven with rigid types. If you really want a hard type system, just move to GO or Rust…

C's type checker is the MMU - when you mess up types, it give you a segfault. If you're lucky.

Funny, but for the readers who don't know C -- this isn't true. C has type checking at compile time.

Re: Python types have an expectations problem

#27

Post by sco1 »

sesgoe wrote: post #8

From my experience working on an older Python codebase, this issue is definitely a headache. It's extremely difficult to gradually adopt typing in an older Python codebase with almost no typing information because the only real "enforcement" option seems to be a CI pipeline running something like `mypy`. This issue compounds in a painful way. Because 99% of your codebase is starting out untyped, you have a couple of…

> you gradually add more and more files to the `mypy` invocation until you're eventually (hopefully) adding entire subfolders, and then maybe eventually the entire codebase

Rather than doing this, which does indeed seem like a headache, it may make more sense to skip import following at the very beginning until your core is typed so you can still enforce typing on the leaf nodes moving forward.

> Eventually you get to `#type: ignore` or `Any`s being thrown around to sidestep the CI pipeline, and your typing story has collapsed again

While there are some cases where this is truly the best option, ultimately you get to the point where you just don't allow this, otherwise what's the point of all the effort?

> and for people on the team not passionate about typing, they never worried about it Asking the entire team to become Python typing gurus to sort out these issues was a non-starter.

The faster the core can be typed (and typed correctly), the easier it becomes for those who are less passionate. Presumably someone has done the calculus to determine that this effort is worthwhile, so while the team doesn't necessarily all have to reach guru level, they need to be convinced to continue the work. Removing barriers is huge for this, since as you've noticed once it starts being easy to ignore it's really challenging to stop ignoring.

Re: Python types have an expectations problem

#28

Post by ringofchaos »

I have been using python library Pydantic with FastApi for input and output data validation for Rest API and it solves some of the problems.

I wanted to check if the use case can be generalized for all situations. For example, the code below will throw a runtime error

* Input should be a valid integer, unable to parse string as an integer [type=int_parsing, input_value='A', input_type=str] *

```

  from pydantic import BaseModel

  class User(BaseModel):
     id: int
     name: str

  def print_user(user: User):
     print(user.name, user.id)

  user_1 = User(id="A", name="John")

  print_user(user_1)
```

I have also been using typescript with React Application and really find it better compared to python due to type checking. But I can still imagine a future when type-checking gets incorporated in native python, unlike javascript/typescript. The groundwork has been laid already with type hints.

Re: Python types have an expectations problem

#29

Post by nickdrozd »

posix_monad wrote: post #11

In Python it's common to have functions whose return types depend on the run-time arguments. For example: def fooify(x): if isinstance(x, list): map(fooify, x) else: x * 2 I guess this is to make scripting more forgiving. In typical typed languages, you would have two functions instead: fooify : number -> number fooifyMany : [number] -> [number] But in the Python community, it's common to have a big function with man…

That's a recursive function, so to make it fully general a recursive type can be used:

  type CompoundInt = int | list[CompoundInt]
  
  def fooify(x: CompoundInt) -> CompoundInt:
      if isinstance(x, int):
          return x * 2
      else:
          return list(map(fooify, x))
  
  print(fooify([1, [3, 4, 5], [6, 7, 8], [[[4, 4, 4]]]]))
This uses the `type` keyword introduced in 3.12. Unfortunately Mypy doesn't support it yet :( so this workaround can be used instead:

  from __future__ import annotations
  
  from typing import TYPE_CHECKING
  
  if TYPE_CHECKING:
      CompoundInt = int | list[CompoundInt]

Re: Python types have an expectations problem

#30

Post by hauntsaninja »

sesgoe wrote: post #8

From my experience working on an older Python codebase, this issue is definitely a headache. It's extremely difficult to gradually adopt typing in an older Python codebase with almost no typing information because the only real "enforcement" option seems to be a CI pipeline running something like `mypy`. This issue compounds in a painful way. Because 99% of your codebase is starting out untyped, you have a couple of…

Lean heavier on a mypy config file. Make heavy use of per-module configuration, in particular, setting per-module ignore_errors = true for modules you’re not yet ready to type check.

See https://mypy.readthedocs.io/en/stable/existing_code.html for some more advice.

Post reply on HN