Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Jupyter is data scientists’ computational notebook of choice

nature.com

Post reply on HN
Print view View original

101–110 of 308 posts

Re: Why Jupyter is data scientists’ computational notebook of choice

#101

Post by enriquto »

dimatura wrote: post #91
enriquto wrote: post #62

I love notebooks as a way to present information, data, code and computations. However, I cannot stand typing any text into a web browser window. Is there any way to edit a jupyter notebook with a text editor and then run it in the browser? The native json is not really human-editable.

I have felt the same way in the past. There are some ways to do this, but none is great. Unfortunately using a text editor to put in the notebook text areas is not that straightforward because of security features in modern browsers. Since jupyter is actually a server (usually running locally) it's possible to communicate directly with it from a sufficiently advanced editor, but haven't seen any good execution of tha…

This is not what I mean. Mine is a problem of file formats, not of interactivity. I want to edit a text file alone, without need for any web browser in my computer. Then I push the notebook to git, and somewhere else it is opened by the browser.

This would be possible today if the notebook file was python code with comments, for example, instead of an uneditable json.

Re: Why Jupyter is data scientists’ computational notebook of choice

#102

Post by blululu »

IMO, Jupyter is nice for presenting the final results of research (like LaTeX), but it is often not the right tool to get there. It's good for professors who teach and publish but it's bad for students to learn and research.

Frankly I find that all programming environments for scientific computing are deficient in some way or another. If you look at the set of features in Visual Studio, R Studio and Jupyter notebooks, you will see that the Union of useful features is large, and the intersection is almost empty.

Re: Why Jupyter is data scientists’ computational notebook of choice

#103

Post by cirgue »

agibsonccc wrote: post #24

The only thing that stops me from being able to use notebooks full time is their intellisense compared to IDEs is horrible. I like being able to use them for demos/presentations, but I can't imagine trying to code within one primarily. Especially when it comes to tracking results. How do people cope with this? Do you supplement it with other tools? I spend a lot of my time in an IDE and then just paste some of the co…

NBextensions and doing mostly data analysis in notebooks then building actual code in a text editor. I would do this even if notebooks had perfect intellisense support.

Re: Why Jupyter is data scientists’ computational notebook of choice

#104

Post by achompas »

makmanalp wrote: post #51

The majority of the complaints I hear about notebooks I think come from a misunderstanding of what they're supposed to be. It's a mashup between a scientific paper and a repl. So it's useful for a bit of both: a) Just like with a paper, you can present scientific or mathematical ideas with accompanying visualizations or simulations. From the REPL side, as a bonus, you get interactivity, and the reader can pause and e…

I agree completely on your first point - notebooks are a poor substitute for proper software tooling. I wrote this recently [1]

> In the case of an analyst, the domain of "software engineering" lies close to their own domain. Projects in both areas require code which (ideally) exhibits clarity and reproducibility. Obfuscated software is bad [...] and idempotency is good.

> The problem, then, is when the analyst takes a core tool from their domain and applies it to a slightly different domain like software engineering. Things go south fast: your notebook has not-quite-imperative code that is untested and unmonitored. It is, in other words, bad software.

As for the point about "refactoring stuff out into python modules as functions," the problem is that the new crop of data scientists aren't learning how to do this. The role of "machine learning engineer" is emerging to address this shortcoming in SWE skill throughout the data science community. It honestly cannot happen quickly enough.

[1] https://buttondown.email/oneshotlearning/archive/c06a0ded-74...

Re: Why Jupyter is data scientists’ computational notebook of choice

#105

Post by achompas »

thousandautumns wrote: post #76
wanderfowl wrote: post #14

I like R for many things, but Python just keeps getting more compelling, particularly given the excellent machine learning packages. As these sorts of toolchain elements get better and better, and as more people realize that there's a benefit to simultaneously training researchers to run code as well as stats, I suspect we'll start to see an exodus from pure R solutions. The real question is when (and whether) new so…

I'm not sure what about Jupyter makes Python more compelling in comparison to R. R is entirely usable in Jupyter Notebooks, and R Notebooks are, in my opinion, possibly superior to Jupyter notebooks in many ways. > and as more people realize that there's a benefit to simultaneously training researchers to run code as well as stats, I suspect we'll start to see an exodus from pure R solutions I'm not sure what you are…

> I would actually argue that most of the Python data science toolchain is years behind what is available in R.

I do not want to litigate this on HN, but the problem with R is the toolchain around your data science work.

You've fit a model in R, and that's great! Now how do you get it into a real-time system? Or how do you test the software you wrote to train the model?

Re: Why Jupyter is data scientists’ computational notebook of choice

#106

Post by enriquto »

thomasfedb wrote: post #73

Jupyter is lovely (and JupyterLab looks delicious ), but the set up required to achieve a reproducible local server with the R kernel and versioned R packages is 100% not. Installing R packages through anaconda is like pulling teeth and the docker images for my Jupyter notebooks push past 6GB and take multiple cups of tea to build. Is there a good solution I'm missing? A good hosted solution perhaps?

you have google colab notebooks: https://colab.research.google.com

Re: Why Jupyter is data scientists’ computational notebook of choice

#107

Post by Tistel »

enriquto wrote: post #62

I love notebooks as a way to present information, data, code and computations. However, I cannot stand typing any text into a web browser window. Is there any way to edit a jupyter notebook with a text editor and then run it in the browser? The native json is not really human-editable.

its not everyones cup of tea, but, emacs can connect to notebooks: https://github.com/millejoh/emacs-ipython-notebook

If I am just poking around I use a Jupiter notebook. If I have to do a lot of prototyping, I use the emacs plugin. So the muscle memory typing works.

I find the whole "notebooks are a revelation!" thing kind of amusing, given that we have had REPLs for a long time. emacs is just a big REPL if you know elisp. But, yeah, ein is great.

Re: Why Jupyter is data scientists’ computational notebook of choice

#108

Post by solomatov »

NegatioN wrote: post #68
solomatov wrote: post #30

Earlier quoted context omitted.

If you miss intellisense, you can try datalore ( https://datalore.io/ ). P.S. Disclaimer: I lead this project at JetBrains, Inc.

Is your plan with this to always have it as what seems like a hosted service? Is it possible to use it as what seems like a drop-in replacement for jupyter notebooks? We have more data then I think would make sense to transfer out of our clusters/datacenter and privacy issues would probably be raised but I would love to use something like this.

>Is your plan with this to always have it as what seems like a hosted service?

We are seriously considering on premises version.

>Is it possible to use it as what seems like a drop-in replacement for jupyter notebooks?

Jupyter import/export will be released soon.

Re: Why Jupyter is data scientists’ computational notebook of choice

#109

Post by ianhowson »

thomasfedb wrote: post #73

Jupyter is lovely (and JupyterLab looks delicious ), but the set up required to achieve a reproducible local server with the R kernel and versioned R packages is 100% not. Installing R packages through anaconda is like pulling teeth and the docker images for my Jupyter notebooks push past 6GB and take multiple cups of tea to build. Is there a good solution I'm missing? A good hosted solution perhaps?

I run https://rnotebook.io -- it's free for the foreseeable future.

The Docker image is 6GB.

Re: Why Jupyter is data scientists’ computational notebook of choice

#110

Post by neuromantik8086 »

makmanalp wrote: post #51

The majority of the complaints I hear about notebooks I think come from a misunderstanding of what they're supposed to be. It's a mashup between a scientific paper and a repl. So it's useful for a bit of both: a) Just like with a paper, you can present scientific or mathematical ideas with accompanying visualizations or simulations. From the REPL side, as a bonus, you get interactivity, and the reader can pause and e…

> When I see stuff around notebooks for "reproducibility", I'm a bit confused in that notebooks often don't specify any guidance on installation and dependencies, let alone things like arguments and options that a regular old script would.

At the core of this, as some others may have already alluded to already, is that many academic scientists have not been socialized to make a distinction between development and production environments. Jupyter notebooks are clearly beneficial for sandboxing and trying out analyses creatively (with many wrong turns) before running "production" analyses, which ideally should be the ones that are reproducible. For many scientific papers, the analysis stops at "I was messing around in SPSS and MATLAB at 3 AM and got this result" without much consideration for reformulating what the researcher did and rewriting code/scripts so that they can be re-run consistently.

Post reply on HN