Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

HRT's Python fork: Leveraging PEP 690 for faster imports

hudsonrivertrading.com

Post reply on HN
Print view View original

11–20 of 103 posts

Re: HRT's Python fork: Leveraging PEP 690 for faster imports

#11

Post by roadside_picnic »

Interviewed with HRT awhile back. While I didn't get past the final round, their Python internals interview (which I did pass) was an absolute blast to prepare for, and required a really deep dive into implementation specific details of CPython around things like exactly how collisions are handled in dict, details about memory management, etc. Pretty much had to spend a few weeks in the CPython source to prep, and was, for me, worth the interview just to really learn what's going on.

For most teams I would be pretty skeptical of a internal Python fork, but the Python devs at HRT really know their stuff.

Re: HRT's Python fork: Leveraging PEP 690 for faster imports

#12

Post by Spivak »

> This process gets dramatically slower for … modules on distributed file systems, modules with slow side-effects

Oh no. Look I'm not saying you're holding it wrong, it's perfectly valid to host your modules on what is presumably NFS as well as having modules with side effects but what if you didn't.

I've been down this road with NFS (and SMB if it matters) and pain is the only thing that awaits you. It seems like they're feeling it. Storing what is spiritually executable code on shared storage was a never ending source of bugs and mysterious performance issues.

Re: HRT's Python fork: Leveraging PEP 690 for faster imports

#15

Post by htrp »

roadside_picnic wrote: post #11

Interviewed with HRT awhile back. While I didn't get past the final round, their Python internals interview (which I did pass) was an absolute blast to prepare for, and required a really deep dive into implementation specific details of CPython around things like exactly how collisions are handled in dict, details about memory management, etc. Pretty much had to spend a few weeks in the CPython source to prep, and wa…

when milliseconds mean millions

Re: HRT's Python fork: Leveraging PEP 690 for faster imports

#16

Post by nly »

roadside_picnic wrote: post #11

Interviewed with HRT awhile back. While I didn't get past the final round, their Python internals interview (which I did pass) was an absolute blast to prepare for, and required a really deep dive into implementation specific details of CPython around things like exactly how collisions are handled in dict, details about memory management, etc. Pretty much had to spend a few weeks in the CPython source to prep, and wa…

I interviewed with them as well. Something like 6-8 interviews only to be told they then, after that, were circulating my CV amongst teams and didn't have a fit for me...

But yes, like you I had a great experience

Re: HRT's Python fork: Leveraging PEP 690 for faster imports

#17

Post by nasretdinov »

I wonder how much can be saved by using a local file system for imports though. In my testing just a mere presense of a home directory on NFS already dramatically slows down imports (by ~10x) due to Python searching for modules in home directory too by default.

Re: HRT's Python fork: Leveraging PEP 690 for faster imports

#18

Post by its-summertime »

zzzeek wrote: post #6

Gonna call this an antipattern. Do you need all those modules imported in every script ? Well then you save nothing on loadup time, the time will be spent regardless. Does every script not need those imports ? Well they shouldn't be importing those things and this small set of top level imports should be curated into a better, more fine grained list (and if you want to write tools, you can certainly identify these pa…

    import argparse
    parser = argparse.ArgumentParser()
    parser.parse_args()
    import requests
Is an annoying bodge that a programmer should not have to think about, as a random example

Re: HRT's Python fork: Leveraging PEP 690 for faster imports

#19

Post by its-summertime »

> There’s also no way to make imports of the form from module import * lazy

I'd say if you see

    from typing import Final
    [...]
    __all__: Final = ("a", "b", "c")
Its probably 99% safe to pull that from a quick run over of the AST (and caching that for the later import if you want to be fancy)

Of course, should one be doing a star import in a proper codebase?

Re: HRT's Python fork: Leveraging PEP 690 for faster imports

#20

Post by theLiminator »

fabioz wrote: post #5

It'd have been really nice to have that PEP in as it'd have helped me not have to write local imports everywhere. As it is, top-level imports IMHO are only meant to be used for modules required to be used in the startup, everything else should be a local import -- getting everyone convinced of that is the main issue though as it really goes against the regular coding of most Python modules (but the time saved to star…

Yeah, imo that's the way that python should've worked in the first place.

Import-time side effects are definitely nasty though and I wonder what the implications on all downstream code would be. Perhaps a lazy import keyword is a better way forward.

Post reply on HN