If only compiled languages with dead code elimination existed...
HRT's Python fork: Leveraging PEP 690 for faster imports
21–30 of 103 posts
Re: HRT's Python fork: Leveraging PEP 690 for faster imports
#22Interviewed with HRT awhile back. While I didn't get past the final round, their Python internals interview (which I did pass) was an absolute blast to prepare for, and required a really deep dive into implementation specific details of CPython around things like exactly how collisions are handled in dict, details about memory management, etc. Pretty much had to spend a few weeks in the CPython source to prep, and wa…
when milliseconds mean millions
If that was the case, why use Python in the first place?
Re: HRT's Python fork: Leveraging PEP 690 for faster imports
#23Earlier quoted context omitted.
when milliseconds mean millions
If that was the case, why use Python in the first place?
Well there's gonna be people writing code who can't do it in say a high performance C/C++ setup. Not professional programmers, but professional .
Sometimes it will be worth the tradeoff to put that person and a programmer together to code up a solution in another language. Sometimes it will be worth it to have the non-programmer write it in Python and then do Herculean things in the background to make it fast enough.
Re: HRT's Python fork: Leveraging PEP 690 for faster imports
#24Earlier quoted context omitted.
If that was the case, why use Python in the first place?
Well there's gonna be people writing code who can't do it in say a high performance C/C++ setup. Not professional programmers, but professional . Sometimes it will be worth the tradeoff to put that person and a programmer together to code up a solution in another language. Sometimes it will be worth it to have the non-programmer write it in Python and then do Herculean things in the background to make it fast enough…
> Sometimes it will be worth it to have the non-programmer write it in Python and then do Herculean things in the background to make it fast enough.
Nim exists, Crystal exists
Re: HRT's Python fork: Leveraging PEP 690 for faster imports
#25Earlier quoted context omitted.
Well there's gonna be people writing code who can't do it in say a high performance C/C++ setup. Not professional programmers, but professional . Sometimes it will be worth the tradeoff to put that person and a programmer together to code up a solution in another language. Sometimes it will be worth it to have the non-programmer write it in Python and then do Herculean things in the background to make it fast enough…
> Sometimes it will be worth it to have the non-programmer write it in Python and then do Herculean things in the background to make it fast enough. Nim exists, Crystal exists
It would not surprise me if some shops end up using less common languages to fill a niche (or hell, invent their own DSL).
But it also wouldn't surprise me if a lot of shops land on python because that's what their hiring pool knows.
Re: HRT's Python fork: Leveraging PEP 690 for faster imports
#26Interviewed with HRT awhile back. While I didn't get past the final round, their Python internals interview (which I did pass) was an absolute blast to prepare for, and required a really deep dive into implementation specific details of CPython around things like exactly how collisions are handled in dict, details about memory management, etc. Pretty much had to spend a few weeks in the CPython source to prep, and wa…
when milliseconds mean millions
Honestly if you're a millisecond too slow you might as well not trade at all. From my own experience with trying to get Python to go fast for crypto trading, you can get it pretty fast using Cython - single digit microseconds on an average AWS instance for a simple linear regression was my proudest moment. They're probably pushing it even faster because nanoseconds are where the money's at. Many HFT firms are down in the double digit nanoseconds, I believe. Maybe lower.
Re: HRT's Python fork: Leveraging PEP 690 for faster imports
#27I really really want lazy imports in Python, it's would be a godsend for CLIs
Libraries for this have always existed, triggering import on first access. The problem was, they would break linters. But that's not an issue anymore with typing.TYPE_CHECKING.
A PEP is very much welcome, but using lazy import libraries is a fairly common, very old, method of speeding things up. My pre PEP 690 code looks like this:
import typing
from lazy import LazyImport
member = LazyImport('my_module.subpackage', 'member')
member1, member2, = LazyImport('my_module', 'member1', 'member2')
if typing.TYPE_CHECKING:
# normal import, for linter/IDE/navigation.
from my_module.subpackage import member
from my_module import member1, member2Re: HRT's Python fork: Leveraging PEP 690 for faster imports
#28While I see the usefulness of lazy imports, it always seemed a bit backward to me for the importer to ask for lazy import, especially if you make it an import keyword rather than a Python flag. Instead I'd expect the modules to declare (and maybe enforce) that they don't have side effects, that way you know they can be lazily imported, and it opens the door for more optimizations, like declaring the module immutable. That links to the performance barrier of Python due to its dynamic nature as discussed in https://news.ycombinator.com/item?id=44809387
Of course that doesn't solve the overhead of finding the modules, but that could be optimized without lazy import, for example by having a way to pre-compute the module locations at install time.
Re: HRT's Python fork: Leveraging PEP 690 for faster imports
#29I really really want lazy imports in Python, it's would be a godsend for CLIs
Well if you use argparse or one of the many argparse wrappers for a moderately complex CLI you end up lazyfing the CLI parser itself because just fully populating the argparse data structures can easily take half a second or more, so with other startup costs you easily end up with "program --help" taking >1s and any CLI parsing error also taking >1s.
Re: HRT's Python fork: Leveraging PEP 690 for faster imports
#30Earlier quoted context omitted.
If that was the case, why use Python in the first place?
Well there's gonna be people writing code who can't do it in say a high performance C/C++ setup. Not professional programmers, but professional . Sometimes it will be worth the tradeoff to put that person and a programmer together to code up a solution in another language. Sometimes it will be worth it to have the non-programmer write it in Python and then do Herculean things in the background to make it fast enough…
The gulf between high performance C/C++ and Python is vast and includes most other programming languages, many of which are friendly to write or can be made friendly to write for a limited domain, with significantly less rocket science needed than making python faster.