HRT's Python fork: Leveraging PEP 690 for faster imports
hudsonrivertrading.com
HRT's Python fork: Leveraging PEP 690 for faster imports
1–10 of 103 posts
Re: HRT's Python fork: Leveraging PEP 690 for faster imports
#2The author interviewed me and talked about this project, so it was cool seeing a blog post posted about it
Re: HRT's Python fork: Leveraging PEP 690 for faster imports
#3Oh, it's a trading firm. That's why they can fund an internal fork of Python... That sounds nice...
Re: HRT's Python fork: Leveraging PEP 690 for faster imports
#4I know few modules that can take seconds to import but would have been nice to hear how much they actually gained?
Also maybe, if this approach could yield stats on if some import was needed or not ?
Re: HRT's Python fork: Leveraging PEP 690 for faster imports
#5It'd have been really nice to have that PEP in as it'd have helped me not have to write local imports everywhere.
As it is, top-level imports IMHO are only meant to be used for modules required to be used in the startup, everything else should be a local import -- getting everyone convinced of that is the main issue though as it really goes against the regular coding of most Python modules (but the time saved to start up apps I work on does definitely make it worth it).
Re: HRT's Python fork: Leveraging PEP 690 for faster imports
#6Gonna call this an antipattern. Do you need all those modules imported in every script ? Well then you save nothing on loadup time, the time will be spent regardless. Does every script not need those imports ? Well they shouldn't be importing those things and this small set of top level imports should be curated into a better, more fine grained list (and if you want to write tools, you can certainly identify these patterns using tooling similar to that which you wrote for LazyImports).
Re: HRT's Python fork: Leveraging PEP 690 for faster imports
#7I thought HRT was a Cpp shop? Is Python used in their main business applications, or more for quants / data scientists?
Re: HRT's Python fork: Leveraging PEP 690 for faster imports
#8Gonna call this an antipattern. Do you need all those modules imported in every script ? Well then you save nothing on loadup time, the time will be spent regardless. Does every script not need those imports ? Well they shouldn't be importing those things and this small set of top level imports should be curated into a better, more fine grained list (and if you want to write tools, you can certainly identify these pa…
There are often large programs where not every invocation imports every module.
The lazy import approach was pioneered in Mercurial I believe, where it cut down startup times by 3x.
Re: HRT's Python fork: Leveraging PEP 690 for faster imports
#9I thought HRT was a Cpp shop? Is Python used in their main business applications, or more for quants / data scientists?
every quant shop has QR and QT people that can barely write passable python let alone cpp - then the QD people have to integrate that stuff with prod cpp pipelines.
Re: HRT's Python fork: Leveraging PEP 690 for faster imports
#10Gonna call this an antipattern. Do you need all those modules imported in every script ? Well then you save nothing on loadup time, the time will be spent regardless. Does every script not need those imports ? Well they shouldn't be importing those things and this small set of top level imports should be curated into a better, more fine grained list (and if you want to write tools, you can certainly identify these pa…
For personal one file utility scripts, I'll sometimes only import a module on a code path that needs it. And make it global if the scope gets in the way.
It's dirty, but speeds things up vs putting all imports at the top.