Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Jupyter is data scientists’ computational notebook of choice

nature.com

Post reply on HN
Print view View original

281–290 of 308 posts

Re: Why Jupyter is data scientists’ computational notebook of choice

#281

Post by kazagistar »

ubernostrum wrote: post #261
gaius wrote: post #259

Earlier quoted context omitted.

They're scientists who increasingly have to do programming as part of their science. And notebooks are a godsend for them 2/3 of scientific research cannot be reproduced by other scientists. But tell us more about why scientists should ignore best practices from other fields.

Show me the reproducible research in programming about the merits of different type systems (murky at best). Or of different approaches to testing. Or software architecture. Or... well, most of the stuff day-to-day working programmers actually do . There are barely even attempts at rigor in most of our practices, let alone the kind of reviewed and reproduced results we demand from the sciences.

I run my unit and integration tests with every build, and they reproducibly pass if my code is working. If you have code, it doesn't take much to make it able to run again and get the same result, and it's frustrating to see Jupyter users mess it up.

Re: Why Jupyter is data scientists’ computational notebook of choice

#282

Post by red-tea »

gaius wrote: post #267
red-tea wrote: post #218

Earlier quoted context omitted.

Why should there be something better? Just because you want it? Someone has to make it.

There already are many better things as many commenters have pointed out

Are there? For python? People like python.

Re: Why Jupyter is data scientists’ computational notebook of choice

#283

Post by setgree »

thomasfedb wrote: post #73

Jupyter is lovely (and JupyterLab looks delicious ), but the set up required to achieve a reproducible local server with the R kernel and versioned R packages is 100% not. Installing R packages through anaconda is like pulling teeth and the docker images for my Jupyter notebooks push past 6GB and take multiple cups of tea to build. Is there a good solution I'm missing? A good hosted solution perhaps?

I work for a company, Code Ocean, that aims to solve this issue. We have a custom-UI for installing packages through a variety of package managers, including Conda, CRAN, etc. Here's an example of some Julia notebooks being rendered to HTML with the environment fully configured and accessible https://codeocean.com/2018/08/16/counterexamples-on-the-mono...

Re: Why Jupyter is data scientists’ computational notebook of choice

#284

Post by yodon »

pavanagrawal123 wrote: post #57
projectramo wrote: post #35

Here are the issues with Jupyter, and most other flavor, of notebook: 1. variables have to be explicitly output The most important tool for programming, for me, is that window that shows you the current state of all the variables. When I step through a program, I look at the state. 90% of my debugging solutions come from seeing that variable doesn't have the right state. 2. Intellisense For the love of god, I do not…

Hey There! I'm trying to solve the issue of IntelliSense.. I'm building/improving Jupyter Notebooks inside VSCode: https://github.com/pavanagrawal123/VSNotebooks . It's a fork from another extension somebody already built, but all activity is dead, so I'm starting up dev on an active fork. I'd love to hear any feedback y'all have! :) Also planning to add some nice debug features, plus hopefully integration into the i…

Microsoft just announced an initiative like this (unfunded, community-based, likely at risk of becoming abandonware), perhaps you could combine your efforts with theirs? (The issue in their code that I'm personally most impacted by in lack of support for conda[0])

[0]https://github.com/lorenzo2897/vscode-ipe/issues/162

Re: Why Jupyter is data scientists’ computational notebook of choice

#285

Post by yodon »

yodon wrote: post #284
pavanagrawal123 wrote: post #57

Earlier quoted context omitted.

Hey There! I'm trying to solve the issue of IntelliSense.. I'm building/improving Jupyter Notebooks inside VSCode: https://github.com/pavanagrawal123/VSNotebooks . It's a fork from another extension somebody already built, but all activity is dead, so I'm starting up dev on an active fork. I'd love to hear any feedback y'all have! :) Also planning to add some nice debug features, plus hopefully integration into the i…

Microsoft just announced an initiative like this (unfunded, community-based, likely at risk of becoming abandonware), perhaps you could combine your efforts with theirs? (The issue in their code that I'm personally most impacted by in lack of support for conda[0]) [0] https://github.com/lorenzo2897/vscode-ipe/issues/162

HN thread for that announcement: https://news.ycombinator.com/item?id=18346198

Re: Why Jupyter is data scientists’ computational notebook of choice

#286

Post by improbable22 »

kazagistar wrote: post #279
analog31 wrote: post #254

Earlier quoted context omitted.

This is kind of a broad observation, but scientists tend to borrow tools from a huge variety of fields, and use them in ways that seem un-disciplined to the practitioners of those fields. For instance, an engineer would be horrified to see me working in the machine shop without a fully dimensioned and toleranced drawing. A project manager would be disturbed to learn that I don't have a pre-written plan for my next ta…

But in the case of dev and prod distinction it has nothing to do with fitting some over-constrained engineering principle, but about fitting actual science: if you cannot reproduce something, you don't have a result, you have a fluke.

I think GP here is an insightful comment. Reproducing things is indeed important, but re-running code is much too narrow a definition, and possibly distractingly narrow.

Maybe your awful notebook gets the same answer you got the day before on the blackboard. Or the same answer your collaborator got independently, perhaps with different tools. Those might be great checks that you understand what you're doing. Spending time on them might be more valuable for finding errors than spending time on making one approach run without human intervention.

Not to say that there aren't some scientists who would benefit from better engineering. But it's too strong to say that fixing everything that looks wrong to engineer's eyes is automatically a good idea.

Re: Why Jupyter is data scientists’ computational notebook of choice

#287

Post by pavanagrawal123 »

yodon wrote: post #284
pavanagrawal123 wrote: post #57

Earlier quoted context omitted.

Hey There! I'm trying to solve the issue of IntelliSense.. I'm building/improving Jupyter Notebooks inside VSCode: https://github.com/pavanagrawal123/VSNotebooks . It's a fork from another extension somebody already built, but all activity is dead, so I'm starting up dev on an active fork. I'd love to hear any feedback y'all have! :) Also planning to add some nice debug features, plus hopefully integration into the i…

Microsoft just announced an initiative like this (unfunded, community-based, likely at risk of becoming abandonware), perhaps you could combine your efforts with theirs? (The issue in their code that I'm personally most impacted by in lack of support for conda[0]) [0] https://github.com/lorenzo2897/vscode-ipe/issues/162

Fixed this in my fork :)

I want to combine, but Neuron has stated they will not be accepting any PRs until December.

Re: Why Jupyter is data scientists’ computational notebook of choice

#288

Post by pavanagrawal123 »

awake wrote: post #248
pavanagrawal123 wrote: post #161

Earlier quoted context omitted.

I'm planning on adding better language support in a couple of weeks! Don't want to limited to Python and R

One thing you may want to be aware of is that the python language server used for vscode runs pylint which performs static analysis on the code. However, Jupyter Notebook uses autocomplete by actually introspecting the variables as they are defined. This creates large differences when doing things such as selecting a column in a pandas dataframe. In jupyter if you press tab on the column name, it can autocomplete and…

yep! this is something I will be focussing on in VSNotebooks. I noticed this as a huge drawback to the current implementation, so I will be fixing it :)

Re: Why Jupyter is data scientists’ computational notebook of choice

#289

Post by rgardaphe »

ssivark wrote: post #112
amirathi wrote: post #65

Version control for Jupyter notebooks was one of the biggest complaint I had. Specifically, diff and merge with the JSON files (.ipynb) is ugly. I built ReviewNb[1] to solve one of those problems (diff). Note that, there is nbdime[2] which works well for local diff/merge. The idea for ReviewNb is to have much tighter integration with GitHub etc. [1] https://reviewnb.com [2] https://nbdime.readthedocs.io/en/latest/

The hard part is that introducing a tool like git (which requires you to choose moments to take a snapshot of the file, and then add some commit message) breaks the flow of interactive experimentation that notebooks are so good for. And then we need to find a way to make those commits useful, because the time ordering of commits could be different from the time order in which cells were run! That is what is crucial t…

This is a super valuable perspective! We (https://qri.io) are building a kind of git/github for datasets and are hoping to talk to would-be users about just this issue. Would love to have your feedback on it (particularly on how commits are registered). Mind if I ping you at the email address you listed? - Rico

Re: Why Jupyter is data scientists’ computational notebook of choice

#290

Post by analog31 »

kazagistar wrote: post #281
ubernostrum wrote: post #261

Earlier quoted context omitted.

Show me the reproducible research in programming about the merits of different type systems (murky at best). Or of different approaches to testing. Or software architecture. Or... well, most of the stuff day-to-day working programmers actually do . There are barely even attempts at rigor in most of our practices, let alone the kind of reviewed and reproduced results we demand from the sciences.

I run my unit and integration tests with every build, and they reproducibly pass if my code is working. If you have code, it doesn't take much to make it able to run again and get the same result, and it's frustrating to see Jupyter users mess it up.

In Jupyter, I do a "restart kernel and run all cells." While it's not to the level of test driven development, it catches the worst of the issues.
Post reply on HN