Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Jupyter is data scientists’ computational notebook of choice

nature.com

Post reply on HN
Print view View original

51–60 of 308 posts

Re: Why Jupyter is data scientists’ computational notebook of choice

#51

Post by makmanalp »

The majority of the complaints I hear about notebooks I think come from a misunderstanding of what they're supposed to be. It's a mashup between a scientific paper and a repl. So it's useful for a bit of both:

a) Just like with a paper, you can present scientific or mathematical ideas with accompanying visualizations or simulations. From the REPL side, as a bonus, you get interactivity, and the reader can pause and experiment with the examples you're giving to improve their understanding or test their hypotheses. If I change this variable, how will the system react? You can just try it!

b) Just like with a REPL, you can type in and execute commands step by step, viewing the output of the previous command instead of running the whole thing at once. From the document side, as a bonus, you get nicer presentation (charts, interactivity, nice and wide sortable tables, etc) than you would in a shell, which comes in handy when doing things like data exploration or mathematical simulation.

It's decidedly NOT there for you to type all your code in like an editor and make a huge mess. It's apples and oranges w.r.t and a poor substitute for something like PyCharm or VS Code or vim. It is there for you to a) try things out yourself, and whatever you discover hopefully eventually make it into proper python modules b) make interesting ideas presentable and explorable for others. That's all!

When I see stuff like "out of order execution is confusing", I don't disagree, but it does make me wonder how long and convoluted the notebooks these people work with are - probably a ripe candidate to refactor stuff out into python modules as functions. When I see stuff around notebooks for "reproducibility", I'm a bit confused in that notebooks often don't specify any guidance on installation and dependencies, let alone things like arguments and options that a regular old script would. In that regard I think it's barely an improvement over .py files lying around. When I hear "how do I import a notebook like a python module", I'm very very scared.

Granted, I've seen huge notebooks that are a mess, so I understand the frustration, but it's not like we all haven't seen the single file of code with 5000 lines and 10 nested layers of conditionals at some point in our lives.

Re: Why Jupyter is data scientists’ computational notebook of choice

#52

Post by solomatov »

cphoover wrote: post #11

For people who prefer to code in JS. There is a similar application called observable notebooks that has recently come out: https://beta.observablehq.com/ It offers some nifty things including, well , observables where cells of the scratchpad can automatically update by observing changes from other cells.

If you want to have interactive computations in Python, http://datalore.io/ has support for this. It feels more or less like Excel with ability to write Python in there.

Re: Why Jupyter is data scientists’ computational notebook of choice

#53

Post by currymj »

lixtra wrote: post #4

What was the earliest of these tools? Mathcad? Mathematica? Maple?

The original IPython notebook was consciously imitating Mathematica.

I don’t think it quite makes sense to compare these notebooks to Knuth’s literate programming. The whole point of that was that you could present things out of order, which is impossible and actually a huge pain point for notebooks.

Re: Why Jupyter is data scientists’ computational notebook of choice

#54

Post by pavanagrawal123 »

agibsonccc wrote: post #24

The only thing that stops me from being able to use notebooks full time is their intellisense compared to IDEs is horrible. I like being able to use them for demos/presentations, but I can't imagine trying to code within one primarily. Especially when it comes to tracking results. How do people cope with this? Do you supplement it with other tools? I spend a lot of my time in an IDE and then just paste some of the co…

Hey There! I'm trying to solve this right now in VSCode's in built editor: https://github.com/pavanagrawal123/VSNotebooks . It's a fork from another extension somebody already built, but all activity is dead, so I'm starting up dev on an active fork. I'd love to hear any feedback y'all have! :)

Re: Why Jupyter is data scientists’ computational notebook of choice

#55

Post by pavanagrawal123 »

renjimen wrote: post #42
agibsonccc wrote: post #24

The only thing that stops me from being able to use notebooks full time is their intellisense compared to IDEs is horrible. I like being able to use them for demos/presentations, but I can't imagine trying to code within one primarily. Especially when it comes to tracking results. How do people cope with this? Do you supplement it with other tools? I spend a lot of my time in an IDE and then just paste some of the co…

That's surprising because I have the opposite experience! Since my first cell is to import all of the libraries I want to use to memory, the intellisense works without fail, regardless of how big the libraries are. Comparing that with my VS Code experience where using intellisense to pull up functions' doc strings takes an age for all but the inbuilt Python libraries.

Hey There! I'm trying to solve this right now in VSCode's in built editor: https://github.com/pavanagrawal123/VSNotebooks . It's a fork from another extension somebody already built, but all activity is dead, so I'm starting up dev on an active fork. I'd love to hear any feedback y'all have! :)

Re: Why Jupyter is data scientists’ computational notebook of choice

#56

Post by bunderbunder »

zimablue wrote: post #29
agibsonccc wrote: post #24

The only thing that stops me from being able to use notebooks full time is their intellisense compared to IDEs is horrible. I like being able to use them for demos/presentations, but I can't imagine trying to code within one primarily. Especially when it comes to tracking results. How do people cope with this? Do you supplement it with other tools? I spend a lot of my time in an IDE and then just paste some of the co…

I do the opposite, my job is kind of bad data engineer/scientist/etl minion so it's a lot of dataframes. Work (and often debug) in jupyter -> open the notebook from pycharm when it's got some completed thoughts and write into a python module + test module, tidying up and adding type annotations. Sometimes doing that multiple times so that the notebook is importing from modules which were originally pulled out of the…

I am very hopeful that JupyterLab will get support for the Language Server Protocol sometime soon. That would make all the difference in the world for me. I'd still have to use a terminal to build and run tests, but I wouldn't be surprised if a test runner comes along fairly quickly after that.

(Relevant issue: https://github.com/jupyterlab/jupyterlab/issues/2163)

Re: Why Jupyter is data scientists’ computational notebook of choice

#57

Post by pavanagrawal123 »

projectramo wrote: post #35

Here are the issues with Jupyter, and most other flavor, of notebook: 1. variables have to be explicitly output The most important tool for programming, for me, is that window that shows you the current state of all the variables. When I step through a program, I look at the state. 90% of my debugging solutions come from seeing that variable doesn't have the right state. 2. Intellisense For the love of god, I do not…

Hey There! I'm trying to solve the issue of IntelliSense.. I'm building/improving Jupyter Notebooks inside VSCode: https://github.com/pavanagrawal123/VSNotebooks . It's a fork from another extension somebody already built, but all activity is dead, so I'm starting up dev on an active fork. I'd love to hear any feedback y'all have! :)

Also planning to add some nice debug features, plus hopefully integration into the inbuilt VSCode debugger!

Re: Why Jupyter is data scientists’ computational notebook of choice

#58

Post by currymj »

saosebastiao wrote: post #44
wanderfowl wrote: post #14

I like R for many things, but Python just keeps getting more compelling, particularly given the excellent machine learning packages. As these sorts of toolchain elements get better and better, and as more people realize that there's a benefit to simultaneously training researchers to run code as well as stats, I suspect we'll start to see an exodus from pure R solutions. The real question is when (and whether) new so…

I'm definitely well in the R camp but keep feeling this nagging pull from Python. Especially for trading...it would be so nice to have a language for both research and production, as right now I translate all my research into scala for production.

I hate to be the stereotypical Julia recommender, but it is made for this use case, more so than Python, which isn’t all that much faster than R if speed matters. (Unless you want to try Cython but that’s a whole bag of worms.)

Re: Why Jupyter is data scientists’ computational notebook of choice

#59

Post by daveFNbuck »

agibsonccc wrote: post #24

The only thing that stops me from being able to use notebooks full time is their intellisense compared to IDEs is horrible. I like being able to use them for demos/presentations, but I can't imagine trying to code within one primarily. Especially when it comes to tracking results. How do people cope with this? Do you supplement it with other tools? I spend a lot of my time in an IDE and then just paste some of the co…

When you're processing a lot of data, it can be expensive to keep re-running your whole script every time you make a change. The notebook keeps the results of your earlier steps in memory when you want to change and re-run a later step.

This is a trade-off between how much code you're writing and how much data you're processing. If you're writing maybe 20 lines of code but you have enough input that it takes several minutes to run, the notebook becomes a clear win for your development process.

Post reply on HN