Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Jupyter is data scientists’ computational notebook of choice

nature.com

Post reply on HN
Print view View original

41–50 of 308 posts

Re: Why Jupyter is data scientists’ computational notebook of choice

#41

Post by MrPowers »

agibsonccc wrote: post #24

The only thing that stops me from being able to use notebooks full time is their intellisense compared to IDEs is horrible. I like being able to use them for demos/presentations, but I can't imagine trying to code within one primarily. Especially when it comes to tracking results. How do people cope with this? Do you supplement it with other tools? I spend a lot of my time in an IDE and then just paste some of the co…

I am a Spark data engineer and spend a lot of time in Scala / Python IDEs & browser notebooks. Databricks lets you package code as JAR / wheel files & attach the binaries to the cluster. I write all the complicated code in tested projects that are checked into GitHub & use the notebooks to invoke the functions and visualize results.

Folks that try to do all programming in notebooks typically drown in complexity and suffer.

Re: Why Jupyter is data scientists’ computational notebook of choice

#42

Post by renjimen »

agibsonccc wrote: post #24

The only thing that stops me from being able to use notebooks full time is their intellisense compared to IDEs is horrible. I like being able to use them for demos/presentations, but I can't imagine trying to code within one primarily. Especially when it comes to tracking results. How do people cope with this? Do you supplement it with other tools? I spend a lot of my time in an IDE and then just paste some of the co…

That's surprising because I have the opposite experience! Since my first cell is to import all of the libraries I want to use to memory, the intellisense works without fail, regardless of how big the libraries are. Comparing that with my VS Code experience where using intellisense to pull up functions' doc strings takes an age for all but the inbuilt Python libraries.

Re: Why Jupyter is data scientists’ computational notebook of choice

#43

Post by mattkrause »

wanderfowl wrote: post #14

I like R for many things, but Python just keeps getting more compelling, particularly given the excellent machine learning packages. As these sorts of toolchain elements get better and better, and as more people realize that there's a benefit to simultaneously training researchers to run code as well as stats, I suspect we'll start to see an exodus from pure R solutions. The real question is when (and whether) new so…

It seems to depend on what you’re doing.

Python definitely has more mindshare for machine learning, and particularly deep learning. However, that’s not all of statistics. For things like mixed-effects modeling, I think R still has a clear lead. There are some python packages (e.g., statsmodels) but R’s lme4 has more features, like custom covariance structures, and virtually every textbook and tutorial currently uses R. I’m actually not sure if I’ve ever actually encountered statsmodels in the wild. PyMCMC is relatively popular, but I think bugs/jags are also more common.

Re: Why Jupyter is data scientists’ computational notebook of choice

#44

Post by saosebastiao »

wanderfowl wrote: post #14

I like R for many things, but Python just keeps getting more compelling, particularly given the excellent machine learning packages. As these sorts of toolchain elements get better and better, and as more people realize that there's a benefit to simultaneously training researchers to run code as well as stats, I suspect we'll start to see an exodus from pure R solutions. The real question is when (and whether) new so…

I'm definitely well in the R camp but keep feeling this nagging pull from Python. Especially for trading...it would be so nice to have a language for both research and production, as right now I translate all my research into scala for production.

Re: Why Jupyter is data scientists’ computational notebook of choice

#45

Post by thealfreds »

renjimen wrote: post #42
agibsonccc wrote: post #24

The only thing that stops me from being able to use notebooks full time is their intellisense compared to IDEs is horrible. I like being able to use them for demos/presentations, but I can't imagine trying to code within one primarily. Especially when it comes to tracking results. How do people cope with this? Do you supplement it with other tools? I spend a lot of my time in an IDE and then just paste some of the co…

That's surprising because I have the opposite experience! Since my first cell is to import all of the libraries I want to use to memory, the intellisense works without fail, regardless of how big the libraries are. Comparing that with my VS Code experience where using intellisense to pull up functions' doc strings takes an age for all but the inbuilt Python libraries.

I'm not a Python dev. Is it not common to just type and let it auto import in the required libraries for you?

Re: Why Jupyter is data scientists’ computational notebook of choice

#46

Post by sueders101 »

agibsonccc wrote: post #24

The only thing that stops me from being able to use notebooks full time is their intellisense compared to IDEs is horrible. I like being able to use them for demos/presentations, but I can't imagine trying to code within one primarily. Especially when it comes to tracking results. How do people cope with this? Do you supplement it with other tools? I spend a lot of my time in an IDE and then just paste some of the co…

I've become a big fan of Hydrogen recently. It's Jupyter notebooks for Atom.

https://nteract.io/atom

Re: Why Jupyter is data scientists’ computational notebook of choice

#47

Post by cphoover »

th0ma5 wrote: post #40
cphoover wrote: post #11

For people who prefer to code in JS. There is a similar application called observable notebooks that has recently come out: https://beta.observablehq.com/ It offers some nifty things including, well , observables where cells of the scratchpad can automatically update by observing changes from other cells.

More of an online service, though.

true true.

Re: Why Jupyter is data scientists’ computational notebook of choice

#48

Post by MrPowers »

Notebooks are great for invoking existing functions and exploring data.

Notebooks aren't ideal for creating functions (standard text editor features are lacking and testing is impossible).

Notebooks encourage an "order dependent variable assignment" programming style without abstractions. Here's what you'll commonly see in a notebook:

val df = spark.read.csv("some_data")

df2 = df.withColumn("clean_name", trim("name"))

df3 = df2.filter("clean_name" === "Mark")

I've found that notebooks are very useful if you write all the complicated code in separate GitHub repos and attach binary executables to the cluster. If you try to write all your logic in notebooks, you'll quickly struggle with order dependent, messy code.

Re: Why Jupyter is data scientists’ computational notebook of choice

#49

Post by cwyers »

johnminter wrote: post #12

RStudio with using Rmarkdown is also popular. Both workflows are language agnostic.

Putting aside the R vs Python question (as as noted in this thread, you can use R in a Jupyter notebook and Python in an RMarkdown notebook), I much prefer RMarkdown notebooks. RMarkdown notebooks are plain text, so you can read them easily in any text editor (which also means they play well with git, unlike Jupyter notebooks).

And it's meant to work with the RStudio IDE, so I get a much more seamless experience going between regular code and notebooks (although this is admittedly a more R-centric benefit, at least until and unless RStudio adds Python support outside of notebooks).

Re: Why Jupyter is data scientists’ computational notebook of choice

#50

Post by Insanity »

thealfreds wrote: post #45
renjimen wrote: post #42

Earlier quoted context omitted.

That's surprising because I have the opposite experience! Since my first cell is to import all of the libraries I want to use to memory, the intellisense works without fail, regardless of how big the libraries are. Comparing that with my VS Code experience where using intellisense to pull up functions' doc strings takes an age for all but the inbuilt Python libraries.

I'm not a Python dev. Is it not common to just type and let it auto import in the required libraries for you?

I don't tend to do so in Python whereas I do in Java.

Maybe due to often importing and naming (something you don't do in Java.)

E.g

    Import matplotlib as plot
Vs Import java.util.
Post reply on HN