Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Jupyter is data scientists’ computational notebook of choice

nature.com

Post reply on HN
Print view View original

271–280 of 308 posts

Re: Why Jupyter is data scientists’ computational notebook of choice

#271

Post by Insanity »

cbkeller wrote: post #199
neuromantik8086 wrote: post #110

Earlier quoted context omitted.

> When I see stuff around notebooks for "reproducibility", I'm a bit confused in that notebooks often don't specify any guidance on installation and dependencies, let alone things like arguments and options that a regular old script would. At the core of this, as some others may have already alluded to already, is that many academic scientists have not been socialized to make a distinction between development and pro…

> many academic scientists have not been socialized to make a distinction between development and production environments Geologist here - definitely true in my field. Nonetheless, while I don't develop in notebooks at all, I do use them for "reproducibility" in a sense -- by putting a bit of dependency info in a github repo along with a .ipynb file, I can do things like this: https://mybinder.org/v2/gh/brenhinkeller…

Don't know if it is something you actively need, but the image did not load for me. (Even after re-evaluating the cell).

The image after: "For example (KJ04-70)"

(I also re-ran the preceding cells).

Re: Why Jupyter is data scientists’ computational notebook of choice

#272

Post by baldfat »

jonnycomputer wrote: post #93
makmanalp wrote: post #51

The majority of the complaints I hear about notebooks I think come from a misunderstanding of what they're supposed to be. It's a mashup between a scientific paper and a repl. So it's useful for a bit of both: a) Just like with a paper, you can present scientific or mathematical ideas with accompanying visualizations or simulations. From the REPL side, as a bonus, you get interactivity, and the reader can pause and e…

If you ever have used an R Notebook written in R-Markdown, then its pretty easy to see why Jupyter Notebooks putting everything in JSON is just... infuriatingly wrong-headed. In an R Notebook, I can see my code, I can see my text, everything is exceedingly simple to understand, and I can edit it in any of the fantastic text editors out there (Jupyter's editor is not among them)

Here is a link to why the R Notebook is a much better format for doing science. I also prefer it since I use GIT.

https://rviews.rstudio.com/2017/03/15/why-i-love-r-notebooks...

This is the reason why it is great.

> 1) Plain text representation

Re: Why Jupyter is data scientists’ computational notebook of choice

#273

Post by LeanderK »

agibsonccc wrote: post #37
LeanderK wrote: post #25

Earlier quoted context omitted.

diving code between models/data-pipelines and experiments. Notebook are used for visualization and telling a story why you tried what for the other team-members.

Yeah but the whole point is "interactive coding". It doesn't feel very interactive when I have to context switch all the time :). I'd prefer something closer to what the lisp folks get to do with the repl where you can scratch out an idea and see it working without leaving your environment.

well, I don't think so. Not everything you do is interactive. Data exploration and basic model selection is, but complex models and more complicated data-pipelines/preprocessing isn't, I think. Tensorflow is the opposite of interactive, even in a notebook.

putting models (in a sense of more complicated models, not just a SVM), data-pipelines, shared visualization-code in a src folders and experimenting in the notebook divides stuff that's interactive by nature from "real" coding. I don't context-switch that much to be honest.

I don't really copy code into cells, because I only experiment there.

Also, what happens if you need to share code between notebooks?

I think notebooks should be simple and explain the experiments and the reasoning behind them to your coworkers. Otherise it's hard to coordinate and learn from each others insights into the data.

Re: Why Jupyter is data scientists’ computational notebook of choice

#274

Post by analog31 »

gaius wrote: post #259
ubernostrum wrote: post #257

Earlier quoted context omitted.

There is also the entire bend where these are taking off in a way that doesn't make sense. It makes perfect sense. Just not to a lot of HN readers. The average HN reader is approaching this from a perspective of "I am a professional programmer who might occasionally dabble in scientific computing, and therefore I hate this thing because it's not a professional programmer's tool designed by and for professional progra…

They're scientists who increasingly have to do programming as part of their science. And notebooks are a godsend for them 2/3 of scientific research cannot be reproduced by other scientists. But tell us more about why scientists should ignore best practices from other fields.

Because I don't have four years to get something done, that doesn't do what I want when I finally get it, if it even works at all, and that I can't fix myself.

Okay, that was extreme, and if you think I was talking about programming, it's because you have a guilty conscience. ;-) It actually applies to all interesting fields -- programming, engineering, management, classical music composition, etc. Those fields don't even know what their best practices are, and acknowledge that things take too long and can't be managed. No manager would say: "Our programmers have best practices, so the work will be done next week." Why should scientists have such faith?

Meanwhile, do you trust Maxwell's Equations, Darwinian evolution, quantum mechanics, etc.? How did we establish the physical constants to mostly better than 8 digits of precision? Science has somehow figured out how to make progress despite the messy business of research.

For me, it's not that I "have" to do programming, but that physical science has been computation driven since before the 1940s. Programming is how I think and work. With apologies to Richelieu, "programming is too important to be left to the programmers."

Re: Why Jupyter is data scientists’ computational notebook of choice

#275

Post by byt143 »

currymj wrote: post #236
fjuerfilis wrote: post #88

Earlier quoted context omitted.

I'd second that. R and Python both have the same pre-LLVM performance issues. I don't expect either R or Python to go away either time soon, nor would I want them to, but I would like to see people moving to things like Julia and Nim, which have the same level of expressivity, but are much more performant. I have difficulty imagining many people saying "I love programming in R and Python, but don't like Julia or Nim.…

I would also add two good reasons to stick with R: RStudio and Hadley Wickham. In theory, there are Python and Julia equivalents to RStudio (JupyterLab, Spyder, PyCharm, Juno, whatever) but RStudio is just so, so, so good. A truly great piece of software. And of course if you have a data pipeline type workflow, and it fits into the Hadleyverse paradigm and isn't too performance intensive, there's nothing better.

Julia has it's own "*verse" type data pipeline framework, with an even greater variety of backends and plotting solutions than R.

It's still in development (mutate and select have PRs) but it's almost there.

https://github.com/queryverse/Query.jl

Re: Why Jupyter is data scientists’ computational notebook of choice

#276

Post by thomasfedb »

kvlr wrote: post #264
thomasfedb wrote: post #73

Jupyter is lovely (and JupyterLab looks delicious ), but the set up required to achieve a reproducible local server with the R kernel and versioned R packages is 100% not. Installing R packages through anaconda is like pulling teeth and the docker images for my Jupyter notebooks push past 6GB and take multiple cups of tea to build. Is there a good solution I'm missing? A good hosted solution perhaps?

I run https://nextjournal.com We archive full-stack reproducibility by allowing you to install arbitrary software and version these environments using docker. You can reuse these in other articles or pull and use them locally. `xoxo` is a signup code you can you if you want to give it a try.

Looks cool, do you have an article that explains the stack from a technical perspective? Is this based on Jupyter?

Re: Why Jupyter is data scientists’ computational notebook of choice

#277

Post by kvlr »

thomasfedb wrote: post #276
kvlr wrote: post #264

Earlier quoted context omitted.

I run https://nextjournal.com We archive full-stack reproducibility by allowing you to install arbitrary software and version these environments using docker. You can reuse these in other articles or pull and use them locally. `xoxo` is a signup code you can you if you want to give it a try.

Looks cool, do you have an article that explains the stack from a technical perspective? Is this based on Jupyter?

Not yet. It isn't based on Jupyter but it's all written in Clojure. Been meaning to do a writeup on our stack for some time…

In addition to our own runtime protocol, we support Jupyter kernels and you can import Jupyter and (R)markdown documents.

Re: Why Jupyter is data scientists’ computational notebook of choice

#278

Post by infinite8s »

bodkan wrote: post #227
azag0 wrote: post #164

I love Jupyter Notebook for experimenting and rapid creation of reports, but dislike it for not being able to use my editor and for intermingling inputs and outputs in a single file. So I'm working on an alternative frontend to Jupyter kernels, which is heavily inspired by KnitR: https://github.com/azag0/knitj It is still being developed, but it's functional and I use it every day.

This looks incredible! Does your project already support other language kernels than the Python kernel? I use R for 90% of my work, but most of it has been happening in Jupyter notebooks (which I'm not a huge fan of, despite practically living in them for the past 4 years of my life). Thanks for sharing!

If you use R so much why do t you use RStudio instead?

Re: Why Jupyter is data scientists’ computational notebook of choice

#279

Post by kazagistar »

analog31 wrote: post #254
neuromantik8086 wrote: post #110

Earlier quoted context omitted.

> When I see stuff around notebooks for "reproducibility", I'm a bit confused in that notebooks often don't specify any guidance on installation and dependencies, let alone things like arguments and options that a regular old script would. At the core of this, as some others may have already alluded to already, is that many academic scientists have not been socialized to make a distinction between development and pro…

This is kind of a broad observation, but scientists tend to borrow tools from a huge variety of fields, and use them in ways that seem un-disciplined to the practitioners of those fields. For instance, an engineer would be horrified to see me working in the machine shop without a fully dimensioned and toleranced drawing. A project manager would be disturbed to learn that I don't have a pre-written plan for my next ta…

But in the case of dev and prod distinction it has nothing to do with fitting some over-constrained engineering principle, but about fitting actual science: if you cannot reproduce something, you don't have a result, you have a fluke.

Re: Why Jupyter is data scientists’ computational notebook of choice

#280

Post by saosebastiao »

currymj wrote: post #58
saosebastiao wrote: post #44

Earlier quoted context omitted.

I'm definitely well in the R camp but keep feeling this nagging pull from Python. Especially for trading...it would be so nice to have a language for both research and production, as right now I translate all my research into scala for production.

I hate to be the stereotypical Julia recommender, but it is made for this use case, more so than Python, which isn’t all that much faster than R if speed matters. (Unless you want to try Cython but that’s a whole bag of worms.)

I've tried it while it was unstable and it was excellent (I remember some random forest training that went from a few days down to half an hour). Since everything was unstable at the time, one update was all it took to break everything. Now that 1.0 is out, I'd definitely like to pick it up again. Unfortunately for my current use cases, the ecosystem just doesn't exist like it does with R or Python.
Post reply on HN