Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Data-Oriented Programming in Python

moderndescartes.com

Post reply on HN
Print view View original

1–10 of 47 posts

Re: Data-Oriented Programming in Python

#3

Post by barefeg »

tomrod wrote: post #2

This is a wonderfully technical article. I'd love to learn more about Python internals as a scientific coder.

I recommend any of the talks by James Powell at PyData. For example this one https://youtu.be/cKPlPJyQrt4

Edit: maybe this one on Numpy may be more relevant: https://youtu.be/u2yvNw49AX4

Re: Data-Oriented Programming in Python

#4

Post by pedrovhb »

tomrod wrote: post #2

This is a wonderfully technical article. I'd love to learn more about Python internals as a scientific coder.

The official Python documentation is excellent, and in many ways goes beyond providing just a list of existing modules and what they do. Sometimes if I'm bored I'll actually just pull up documentation for something I'm not 100% familiar with and have a look around, and I almost always find something new and useful. A couple of interesting ones are [0][1], and [2] is a nice starting point for discovering more. Not everyone's cup of tea, but I also found it enjoyable dive into asyncio with the docs.

[0] https://docs.python.org/3/howto/descriptor.html [1] https://docs.python.org/3/library/collections.html [2] https://docs.python.org/3/

Re: Data-Oriented Programming in Python

#7

Post by duped »

I'm curious how you would do data oriented programming in a language with no type system and no control over memory layout. And I guess the answer is "you can't, but JITs might exist someday that do it for you"

But you can't wave your hands around and say compiler optimizations will fix performance problems - they can, but they're not magic, and the arrow in the proverbial knee for optimization passes are language semantics that make them impossible to realize (forcing the authors to either abandon the passes, or rely on things like dynamic deoptimization which is not free).

Re: Data-Oriented Programming in Python

#9

Post by sirwhinesalot »

duped wrote: post #7

I'm curious how you would do data oriented programming in a language with no type system and no control over memory layout. And I guess the answer is "you can't, but JITs might exist someday that do it for you" But you can't wave your hands around and say compiler optimizations will fix performance problems - they can, but they're not magic, and the arrow in the proverbial knee for optimization passes are language se…

By using only coding patterns that are known to JIT well and lower level primitive types and containers if provided by the language. Maximizing the use of packages written in native code also helps.

The resulting code is even more annoying to write than using a lower level language typed language in the first place, but ecosystem access sometimes makes up for it.

Hopefully tools like mypyc get better, letting well-typed python code with reasonable usage patterns be compiled to reasonably efficient native code.

Last time I used it I was pleased with the performance benefits but it couldn't even compile all files in a module to a single shared library, despite this being mentioned as possible (and recommended) in the docs. Maybe I was doing something wrong, but they don't answer their github issues often, alas.

Any little thing helps though, it's one thing for throwaway scripts to be inefficient, but applications? At a large scale it is a monstrous waste of time and literal energy.

Post reply on HN