Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Data-Oriented Programming in Python

moderndescartes.com

Post reply on HN
Print view View original

31–40 of 47 posts

Re: Data-Oriented Programming in Python

#31

Post by bombolo »

cauch wrote: post #16

It's a details, but I keep seeing it: > Yet, [the scientists] struggle to move away from Python, because of network effects, and because Python’s beginner-friendliness is appealing to scientists for whom programming is not a first language. I don't believe it's the whole story. In my case, during my 13 years in academia, I saw my field going away from C++ and towards python. Not because of network effects (it was the…

> Another aspect is that you will present your scientific results to your colleagues, not your code (they are not interested in that),

Ah, yes you do not present your code and that's a flaw of science. Because nobody can really reproduce your results without, but it seems modern science gave up on the whole reproducibility thing.

> they are not interested in that

Ah no… they are very much interested in that, but you don't give them out because you want to keep adding little details in the form of new papers, rather than letting other groups compete on the same research. So keeping the code proprietary makes it difficult for other groups to do better than you.

It's all very unscientific and uncooperative really. Let's not pretend it's "science"…

Re: Data-Oriented Programming in Python

#32

Post by periheli0n »

Helmut10001 wrote: post #30
cauch wrote: post #16

It's a details, but I keep seeing it: > Yet, [the scientists] struggle to move away from Python, because of network effects, and because Python’s beginner-friendliness is appealing to scientists for whom programming is not a first language. I don't believe it's the whole story. In my case, during my 13 years in academia, I saw my field going away from C++ and towards python. Not because of network effects (it was the…

This is spot on, thank you for this summary! I just finished working on an exploratory data science paper that was in the making for the last 5 years. In the beginning, I set the goal to version the whole process in git. It was incredibly difficult to not accumulate an immense mountain of tech debt during this whole time. We had 4 internal review rounds (5 researchers from two universities), then 2 immediate rejectio…

> The paper now consists of 10 Jupyter Notebooks and about 16000 lines of code with 1200 git revisions

I‘ve been there too! And about 1 year after the paper was finally published, fresh starters in my lab couldn’t run the code anymore because of an obscure Pandas error caused by a version change. Then Python 2.7 was shelved and created more problems.

In conclusion the considerable effort I put in to create Jupyter notebooks recreating every single figure from the paper was not worth it.

Re: Data-Oriented Programming in Python

#33

Post by Helmut10001 »

periheli0n wrote: post #32
Helmut10001 wrote: post #30

Earlier quoted context omitted.

This is spot on, thank you for this summary! I just finished working on an exploratory data science paper that was in the making for the last 5 years. In the beginning, I set the goal to version the whole process in git. It was incredibly difficult to not accumulate an immense mountain of tech debt during this whole time. We had 4 internal review rounds (5 researchers from two universities), then 2 immediate rejectio…

> The paper now consists of 10 Jupyter Notebooks and about 16000 lines of code with 1200 git revisions I‘ve been there too! And about 1 year after the paper was finally published, fresh starters in my lab couldn’t run the code anymore because of an obscure Pandas error caused by a version change. Then Python 2.7 was shelved and created more problems. In conclusion the considerable effort I put in to create Jupyter no…

I created Docker Container Images for every versioned "release" of the paper that we maintain in our Gitlab Registry (a CI builds those images automatically from `.gitlab-ci.yml`, `docker-compose.yml` using `docker:dind`), so you can pull a specific Docker Image for every version of the paper that will definitely have the correct dependencies (because the Jupyter Notebooks were tested with each specific version), including Jupyter itself.

Re: Data-Oriented Programming in Python

#34

Post by ogarten »

Helmut10001 wrote: post #33
periheli0n wrote: post #32

Earlier quoted context omitted.

> The paper now consists of 10 Jupyter Notebooks and about 16000 lines of code with 1200 git revisions I‘ve been there too! And about 1 year after the paper was finally published, fresh starters in my lab couldn’t run the code anymore because of an obscure Pandas error caused by a version change. Then Python 2.7 was shelved and created more problems. In conclusion the considerable effort I put in to create Jupyter no…

I created Docker Container Images for every versioned "release" of the paper that we maintain in our Gitlab Registry (a CI builds those images automatically from `.gitlab-ci.yml`, `docker-compose.yml` using `docker:dind`), so you can pull a specific Docker Image for every version of the paper that will definitely have the correct dependencies (because the Jupyter Notebooks were tested with each specific version), inc…

That is awesome and should be the minimum standard, but even just getting the entire infrastructure set up is something almost all scientists are not willing to do.

In my experience, one should be happy if code is version controlled in a proper way. Too often, it isn't. In ML this might be a little different but in my field at least (electrical engineering) this is not the case at all.

Re: Data-Oriented Programming in Python

#35

Post by Helmut10001 »

ogarten wrote: post #34
Helmut10001 wrote: post #33

Earlier quoted context omitted.

I created Docker Container Images for every versioned "release" of the paper that we maintain in our Gitlab Registry (a CI builds those images automatically from `.gitlab-ci.yml`, `docker-compose.yml` using `docker:dind`), so you can pull a specific Docker Image for every version of the paper that will definitely have the correct dependencies (because the Jupyter Notebooks were tested with each specific version), inc…

That is awesome and should be the minimum standard, but even just getting the entire infrastructure set up is something almost all scientists are not willing to do. In my experience, one should be happy if code is version controlled in a proper way. Too often, it isn't. In ML this might be a little different but in my field at least (electrical engineering) this is not the case at all.

Yes, I know and I was pretty stubborn with my initial goals to go through with this.. (probably will also cost my employment, but I learned a lot and it was thus still worth it).

Re: Data-Oriented Programming in Python

#37

Post by cauch »

bombolo wrote: post #31
cauch wrote: post #16

It's a details, but I keep seeing it: > Yet, [the scientists] struggle to move away from Python, because of network effects, and because Python’s beginner-friendliness is appealing to scientists for whom programming is not a first language. I don't believe it's the whole story. In my case, during my 13 years in academia, I saw my field going away from C++ and towards python. Not because of network effects (it was the…

> Another aspect is that you will present your scientific results to your colleagues, not your code (they are not interested in that), Ah, yes you do not present your code and that's a flaw of science. Because nobody can really reproduce your results without, but it seems modern science gave up on the whole reproducibility thing. > they are not interested in that Ah no… they are very much interested in that, but you…

> but it seems modern science gave up on the whole reproducibility thing.

It looks like you don't understand the definition and the point of reproducibility. If you just click on a button to recompute the results from the same code, this is not "reproducibility". The point of reproducibility is to have other scientists able to confirm your results in a fully independent way. They should be able to do it from scratch, without your code, and reach the same conclusion.

Using the code is therefore against reproducibility: people who use the code are not reproducing the experiment.

It is not restricted to the code. At CERN, the Atlas experiment and the CMS experiment are designed to be able to observe the same phenomenons, in order to bring independent measurements. Because of the notion of independence and reproducibility, while working in the same building, there are strong instructions to not share any non-published details with the members of the other experiment. The reason is that if there is an error in the reasoning, or if we observe a statistical fluctuations, or if there is a bug in the code, when the other experiment comes up with their results, their results will be unbiased.

Additionally, it is good practice to have several team doing their analysis in parallel on the same subject, each using their own code. This is called "cross-check analyses" and is a very important scientific procedure that is able to discover code bugs way more efficiently than unit test or code review.

> but you don't give them out

Why do you say we don't give them out? The code is available to reviewers. It's on the collaboration git repository and anyone in the collaboration can check it out.

They are not interested in that the same way they are not interested in the schematic of the Intel processor that was used in the laptop of the person who run the algorithm.

They are, however, interested in understanding deeply the reasoning and the logic that led to the conclusion, to the point that, if they want, they can rewrite their own code (in any language they want) and reach the same result. They don't see the code (not because they can't, but because they are smart enough to understand what reproducibility means), but they know what the code is doing.

> It's all very unscientific and uncooperative really. Let's not pretend it's "science"…

You clearly have no idea what you are talking about. Therefore, your conclusions are worthless.

Re: Data-Oriented Programming in Python

#38

Post by [deleted] »

cauch wrote: post #37
bombolo wrote: post #31

Earlier quoted context omitted.

> Another aspect is that you will present your scientific results to your colleagues, not your code (they are not interested in that), Ah, yes you do not present your code and that's a flaw of science. Because nobody can really reproduce your results without, but it seems modern science gave up on the whole reproducibility thing. > they are not interested in that Ah no… they are very much interested in that, but you…

> but it seems modern science gave up on the whole reproducibility thing. It looks like you don't understand the definition and the point of reproducibility. If you just click on a button to recompute the results from the same code, this is not "reproducibility". The point of reproducibility is to have other scientists able to confirm your results in a fully independent way. They should be able to do it from scratch,…

[deleted]

Re: Data-Oriented Programming in Python

#39

Post by sacado2 »

cauch wrote: post #37
bombolo wrote: post #31

Earlier quoted context omitted.

> Another aspect is that you will present your scientific results to your colleagues, not your code (they are not interested in that), Ah, yes you do not present your code and that's a flaw of science. Because nobody can really reproduce your results without, but it seems modern science gave up on the whole reproducibility thing. > they are not interested in that Ah no… they are very much interested in that, but you…

> but it seems modern science gave up on the whole reproducibility thing. It looks like you don't understand the definition and the point of reproducibility. If you just click on a button to recompute the results from the same code, this is not "reproducibility". The point of reproducibility is to have other scientists able to confirm your results in a fully independent way. They should be able to do it from scratch,…

> It looks like you don't understand the definition and the point of reproducibility. If you just click on a button to recompute the results from the same code, this is not "reproducibility". The point of reproducibility is to have other scientists able to confirm your results in a fully independent way. They should be able to do it from scratch, without your code, and reach the same conclusion.

I highly disagree with that. In machine learning for instance, it's very important to be able to check authors' claims because otherwise you can't know if they are lying or not, or if they cheated or not ("what do you mean we can't use our test data to improve our training process?")

Plus, you can't improve on what others did if they don't let you access their own code and dataset. "We got better results than previous works it seems, but we can't know for sure because maybe that's because our method is better, maybe that's because our code is better, we will never know." My ML approach is more efficient than yours because you wrote it in python on your laptop and I wrote it in C and spent a lot of time to optimize it, plus I have a faster computer. Yeah sure. That's not science.

Re: Data-Oriented Programming in Python

#40

Post by cauch »

sacado2 wrote: post #39
cauch wrote: post #37

Earlier quoted context omitted.

> but it seems modern science gave up on the whole reproducibility thing. It looks like you don't understand the definition and the point of reproducibility. If you just click on a button to recompute the results from the same code, this is not "reproducibility". The point of reproducibility is to have other scientists able to confirm your results in a fully independent way. They should be able to do it from scratch,…

> It looks like you don't understand the definition and the point of reproducibility. If you just click on a button to recompute the results from the same code, this is not "reproducibility". The point of reproducibility is to have other scientists able to confirm your results in a fully independent way. They should be able to do it from scratch, without your code, and reach the same conclusion. I highly disagree wit…

What you are talking about is important, but it is not what corresponds to the notion of "reproducibility in science".

Ideally, what we need is being able to 1) reproduce from scratch, 2) rerun the experiment with the code used (which has a lot of pragmatical advantages too).

In fact, I would argue that even the examples you are giving are not "reproducibility". Reproducibility means that if you redo it _from scratch_, you reach the same conclusion. If the conclusion is "this algorithm describe in this paper is 20+-10% better than the algorithm described in this other paper, in those conditions", then, ideally, what you need to do is to read the first paper and follow the explanation to recreate the algorithm, read the second paper and follow the explanation to recreate the algorithm, read the conditions and reproduce them yourself, and then, you check if you get a compatible conclusion.

As for improving on top of others, again, the scientific paper explains the algorithm, just apply the recipe in your personal implementation, and you will have the same result. Even better: if you don't, you may have found a bug that exists in their implementation, and you have made a scientific progress that you would not have obtained if you would have just cut-and-pasted their code.

But as I've said, the ideal is to have both. Still, the sentence that we reacted on as "not scientific because not compatible with reproducibility" is still valid. It is true that, in practice, the code is not very useful in science (it may be a bit useful, but not as crucial as some may think), and it is true that this does not lead to a reproducibility problem, because the goal of the concept of reproducibility is to guarantee the existence of _independent_ implementations to check if we reach the same conclusion.

Post reply on HN