Live data from Hacker News

Data-Oriented Programming in Python

moderndescartes.com

21–30 of 47 posts

Re: Data-Oriented Programming in Python

#21
post #19

Earlier quoted context omitted.

Computer scientists also assume that you know what inputs your program needs and what is the range of the outputs. That is out of touch with scientific research. We may change overnight completely the inputs, the core logic and the outputs. Having to babysit function signatures, manage memory and types throughout these activities is just draining.

That is also how "computer scientists" and software engineers work. Our time is just valued a lot higher so we've come up with techniques to make our work more efficient and faster, like structuring our code well using types and function signatures. The added bonus is you get science that's you know, repeatable. Because the difference between industrial code and prototype code is that it gets run so often there can't…

I used to be a software engineer, and now I'm a data scientist. Not exactly the same thing as a real scientist, but I suspect that we have common cause in this area.

One of the hard lessons in the transition was realizing that things that allowed me to work more efficiently when I was a software engineer instead reduce my efficiency in my new career.

You might get a decent analogy of the difference by comparing photographs of the first transistor with pictures of every subsequent transistor. The first transistor's clearly going to be terrible in any production application. But the same characteristics that make it so terrible for practical use were also, to varying degrees, essential to or characteristic of the exploratory process that led to its creation.

It's similar for my R&D code. In order to do my R&D work more efficiently and effectively, I need to do things that would be unholy in production code. This is why there's a separate and essential productionizing step where my output is heavily revamped and possibly even completely rewritten in a different programming language.

re: repeatability, I've discovered that it, too, means something slightly different in a science context than it does in an engineering context.

Re: Data-Oriented Programming in Python

#22
post #7

I'm curious how you would do data oriented programming in a language with no type system and no control over memory layout. And I guess the answer is "you can't, but JITs might exist someday that do it for you" But you can't wave your hands around and say compiler optimizations will fix performance problems - they can, but they're not magic, and the arrow in the proverbial knee for optimization passes are language se…

> I'm curious how you would do data oriented programming in a language with no type system and no control over memory layout.

I'm not sure what language you're referring to? Neither of those is true of Python.

Re: Data-Oriented Programming in Python

#23
post #7

I'm curious how you would do data oriented programming in a language with no type system and no control over memory layout. And I guess the answer is "you can't, but JITs might exist someday that do it for you" But you can't wave your hands around and say compiler optimizations will fix performance problems - they can, but they're not magic, and the arrow in the proverbial knee for optimization passes are language se…

> I'm curious how you would do data oriented programming in a language with no type system and no control over memory layout. I'm not sure what language you're referring to? Neither of those is true of Python.

I'm not sure what either you or the parent consider a proper type system, but the difference between Python and (say) Java's type system is night and day.

If you using the Typing module you might be able to get a linter to check things for you, but that's a far cry from what other typed languages have.

Re: Data-Oriented Programming in Python

#24
post #18

Earlier quoted context omitted.

Unfortunately, the terms "data-oriented design " and "data-oriented programming " refer to two different styles of programming. Data-oriented design is the approach to programming made popular by Mike Acton's CppCon keynote—as you say, it focuses on the layout of objects in memory to make the processing of data take advantage of the underlying hardware. Data-oriented programming is a style of programming that, as far…

Wait, what? Thanks for pointing this out. I was pretty close to submitting an order with a Data-oriented Programming book in my cart.

There is no book on data oriented design as far as I know. It would be great if someone could take Mike Actons talk and similar talks and condense the ideas into a book filled with real world examples.

Re: Data-Oriented Programming in Python

#25
post #24
post #18

Earlier quoted context omitted.

Wait, what? Thanks for pointing this out. I was pretty close to submitting an order with a Data-oriented Programming book in my cart.

There is no book on data oriented design as far as I know. It would be great if someone could take Mike Actons talk and similar talks and condense the ideas into a book filled with real world examples.

https://www.dataorienteddesign.com/dodmain/

There is this one. It has been posted here a few times.

EDIT: I'd meant to link to the updated one:

https://www.dataorienteddesign.com/dodbook/

Re: Data-Oriented Programming in Python

#26
post #16

It's a details, but I keep seeing it: > Yet, [the scientists] struggle to move away from Python, because of network effects, and because Python’s beginner-friendliness is appealing to scientists for whom programming is not a first language. I don't believe it's the whole story. In my case, during my 13 years in academia, I saw my field going away from C++ and towards python. Not because of network effects (it was the…

It's nice to be able to collect your data and then do analysis on it in one language. Data work is mostly about getting your data in a place where you can work with it. Rarely are you handed a dataset that is shiny and ready for analysis or regression ...

Python also has the benefit of tooling, where you can use tools you're familiar with and still work on most codebases.

Re: Data-Oriented Programming in Python

#27
post #23

Earlier quoted context omitted.

> I'm curious how you would do data oriented programming in a language with no type system and no control over memory layout. I'm not sure what language you're referring to? Neither of those is true of Python.

I'm not sure what either you or the parent consider a proper type system, but the difference between Python and (say) Java's type system is night and day. If you using the Typing module you might be able to get a linter to check things for you, but that's a far cry from what other typed languages have.

How does that have anything to do with data?

Re: Data-Oriented Programming in Python

#28
post #23

Earlier quoted context omitted.

> I'm curious how you would do data oriented programming in a language with no type system and no control over memory layout. I'm not sure what language you're referring to? Neither of those is true of Python.

I'm not sure what either you or the parent consider a proper type system, but the difference between Python and (say) Java's type system is night and day. If you using the Typing module you might be able to get a linter to check things for you, but that's a far cry from what other typed languages have.

It is night and day. More specifically:

Python's type discipline is stronger, but dynamic.

Java's type discipline is static, but weaker.

Both are most certainly typed. An untyped language would be one like Forth or most assembly languages.

Re: Data-Oriented Programming in Python

#30
post #16

It's a details, but I keep seeing it: > Yet, [the scientists] struggle to move away from Python, because of network effects, and because Python’s beginner-friendliness is appealing to scientists for whom programming is not a first language. I don't believe it's the whole story. In my case, during my 13 years in academia, I saw my field going away from C++ and towards python. Not because of network effects (it was the…

This is spot on, thank you for this summary!

I just finished working on an exploratory data science paper that was in the making for the last 5 years. In the beginning, I set the goal to version the whole process in git. It was incredibly difficult to not accumulate an immense mountain of tech debt during this whole time. We had 4 internal review rounds (5 researchers from two universities), then 2 immediate rejections ("not fitting the scope of journal"), which I considered Major Revisions (because you need to significantly adjust the paper to each new journal) and at the final journal now 1 major revision, maybe another one (but I am not sure yet). All in all, there were about 7 Major Revisions, each taking about 3 to 6 months of 8 hour shifts.

The paper now consists of 10 (chained) Jupyter Notebooks and about 16000 lines of code with 1200 git revisions (the notebooks are versioned in Markdown using Jupytext). I cannot account for the fundamental changes that were necessary due to all the external input - this is really very difficult to plan for in the beginning, when you set out for such a project. I hear my IT colleague arguing "you need tests for all of this!" and then I face another day where I need to write 25 new functions in 6 hours, while preparing lectures, workshops, assess master theses etc. - impossible to me, without the flexibility of python. It is at least pretty different to how things work in production at companies, where you have at least some clear goals/rules.

Post reply on HN