Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Jupyter is data scientists’ computational notebook of choice

nature.com

Post reply on HN
Print view View original

61–70 of 308 posts

Re: Why Jupyter is data scientists’ computational notebook of choice

#61

Post by makmanalp »

projectramo wrote: post #35

Here are the issues with Jupyter, and most other flavor, of notebook: 1. variables have to be explicitly output The most important tool for programming, for me, is that window that shows you the current state of all the variables. When I step through a program, I look at the state. 90% of my debugging solutions come from seeing that variable doesn't have the right state. 2. Intellisense For the love of god, I do not…

Re #2, if you haven't tried the newest versions recently (and especially with the jupyterlab beta which has a nicer completion GUI), I'd encourage you to take a look! It's come a long way, along with the library that's doing the completions under the hood.

Re: Why Jupyter is data scientists’ computational notebook of choice

#62

Post by enriquto »

I love notebooks as a way to present information, data, code and computations.

However, I cannot stand typing any text into a web browser window. Is there any way to edit a jupyter notebook with a text editor and then run it in the browser? The native json is not really human-editable.

Re: Why Jupyter is data scientists’ computational notebook of choice

#63

Post by pavanagrawal123 »

zimablue wrote: post #22

I kind of find Jupyter an indictment of other coding tools really, it's 2018 and they're normally kind of weak or kind of unprogrammable. Feel like we're waiting for someone to really reinvent Emacs, preferably using web tech. Most editors can't open a terminal that you can use VIM keybindings on to search/navigate history and treate like any other buffer. VSCode -> not currently possible because they wrote it in a r…

Hey There! I'm trying to solve this right now in VSCode's in built editor: https://github.com/pavanagrawal123/VSNotebooks . It's a fork from another extension somebody already built, but all activity is dead, so I'm starting up dev on an active fork. I'd love to hear any feedback y'all have! :)

Re: Why Jupyter is data scientists’ computational notebook of choice

#64

Post by short_sells_poo »

daveFNbuck wrote: post #59
agibsonccc wrote: post #24

The only thing that stops me from being able to use notebooks full time is their intellisense compared to IDEs is horrible. I like being able to use them for demos/presentations, but I can't imagine trying to code within one primarily. Especially when it comes to tracking results. How do people cope with this? Do you supplement it with other tools? I spend a lot of my time in an IDE and then just paste some of the co…

When you're processing a lot of data, it can be expensive to keep re-running your whole script every time you make a change. The notebook keeps the results of your earlier steps in memory when you want to change and re-run a later step. This is a trade-off between how much code you're writing and how much data you're processing. If you're writing maybe 20 lines of code but you have enough input that it takes several…

So does the standard terminal repl in python. You can achieve the same workflow with having a plain old python file, and then just use your favorite editor's "Send block of code to console" function. This way, you retain your editor's functionality while you can work just as interactively as with a notebook.

Re: Why Jupyter is data scientists’ computational notebook of choice

#65

Post by amirathi »

Version control for Jupyter notebooks was one of the biggest complaint I had. Specifically, diff and merge with the JSON files (.ipynb) is ugly.

I built ReviewNb[1] to solve one of those problems (diff). Note that, there is nbdime[2] which works well for local diff/merge. The idea for ReviewNb is to have much tighter integration with GitHub etc.

[1] https://reviewnb.com

[2] https://nbdime.readthedocs.io/en/latest/

Re: Why Jupyter is data scientists’ computational notebook of choice

#66

Post by trop »

Question/idea: Could a notebook-model supplant bespoke photographing processing software such as the "darkroom" mode of Lightroom (or darktable). The extant programs essentially take a lot of data (camera's raw output) and apply a configurable recipe to produce intelligible output (an image). Each recipe (stored as an XMP sidecar) is essentially a list of math operations (increase brightness, wavelet decompose, change color model, etc.) and their parameters.

Obviously a great part of why we use Lightroom/darktable is because of the speed with which the recipe-processing occurs. Plus a smooth UI, a catalog-viewing feature, and a well vetted choice of image operations. The appeal of moving this work to a notebook would be that an actively maintained Jupyter ecosystem could supplant lock-in to a specific software, and open up the underlying math magic.

At the very least, this could be an interesting platform for experimenting with image processing methods. And the reordering of cells could become a virtue, to run an image processing pipeline out of the standard order.

I'm curious if anyone has already worked along these lines. I find through a quick web search that people are doing some image processing, but more in the face detection or ML for medical imaging aspects. I see as a basic toolkit that http://scikit-image.org/docs/dev/auto_examples/ is something, though this isn't the whole range of operations needed for, say, fine art image tuning.

Re: Why Jupyter is data scientists’ computational notebook of choice

#67

Post by agibsonccc »

MrPowers wrote: post #41
agibsonccc wrote: post #24

The only thing that stops me from being able to use notebooks full time is their intellisense compared to IDEs is horrible. I like being able to use them for demos/presentations, but I can't imagine trying to code within one primarily. Especially when it comes to tracking results. How do people cope with this? Do you supplement it with other tools? I spend a lot of my time in an IDE and then just paste some of the co…

I am a Spark data engineer and spend a lot of time in Scala / Python IDEs & browser notebooks. Databricks lets you package code as JAR / wheel files & attach the binaries to the cluster. I write all the complicated code in tested projects that are checked into GitHub & use the notebooks to invoke the functions and visualize results. Folks that try to do all programming in notebooks typically drown in complexity and s…

Yeah I agree. We do something similar if we're using zeppelin or beaker. I organize it, put an uber jar in there and then run everything from there. That's a ton easier.

Re: Why Jupyter is data scientists’ computational notebook of choice

#68

Post by NegatioN »

solomatov wrote: post #30
agibsonccc wrote: post #24

The only thing that stops me from being able to use notebooks full time is their intellisense compared to IDEs is horrible. I like being able to use them for demos/presentations, but I can't imagine trying to code within one primarily. Especially when it comes to tracking results. How do people cope with this? Do you supplement it with other tools? I spend a lot of my time in an IDE and then just paste some of the co…

If you miss intellisense, you can try datalore ( https://datalore.io/ ). P.S. Disclaimer: I lead this project at JetBrains, Inc.

Is your plan with this to always have it as what seems like a hosted service?

Is it possible to use it as what seems like a drop-in replacement for jupyter notebooks?

We have more data then I think would make sense to transfer out of our clusters/datacenter and privacy issues would probably be raised but I would love to use something like this.

Re: Why Jupyter is data scientists’ computational notebook of choice

#69

Post by agibsonccc »

thealfreds wrote: post #45
renjimen wrote: post #42

Earlier quoted context omitted.

That's surprising because I have the opposite experience! Since my first cell is to import all of the libraries I want to use to memory, the intellisense works without fail, regardless of how big the libraries are. Comparing that with my VS Code experience where using intellisense to pull up functions' doc strings takes an age for all but the inbuilt Python libraries.

I'm not a Python dev. Is it not common to just type and let it auto import in the required libraries for you?

Java's tooling for this is among the top. We're spoiled compared to the dynamically typed languages :)

Re: Why Jupyter is data scientists’ computational notebook of choice

#70

Post by agibsonccc »

daveFNbuck wrote: post #59
agibsonccc wrote: post #24

The only thing that stops me from being able to use notebooks full time is their intellisense compared to IDEs is horrible. I like being able to use them for demos/presentations, but I can't imagine trying to code within one primarily. Especially when it comes to tracking results. How do people cope with this? Do you supplement it with other tools? I spend a lot of my time in an IDE and then just paste some of the co…

When you're processing a lot of data, it can be expensive to keep re-running your whole script every time you make a change. The notebook keeps the results of your earlier steps in memory when you want to change and re-run a later step. This is a trade-off between how much code you're writing and how much data you're processing. If you're writing maybe 20 lines of code but you have enough input that it takes several…

You can generally persist the results your self to disk though. Especially since a lot of things end up being numpy arrays. So you run 1 script that saves all the results, and another that loads it and runs just the part of your workflow you want. Bonus: it's persisted to disk on top of that! I know things get more complicated than that, but I'd say the compelling use case for notebooks isn't the state saving but more the whole package in one place (state persistence,visualization, interactive repl,..)
Post reply on HN