Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What’s wrong with computational notebooks?

web.eecs.utk.edu

Post reply on HN
Print view View original

171–180 of 223 posts

Re: What’s wrong with computational notebooks?

#171

Post by comment_guy »

vnpc1 wrote: post #110
comment_guy wrote: post #59

I don't get why anyone one who knows how to use an IDE would ever use a notebook, the coding experience is garbage in comparison. I understand they started as a way to get STEM kids coding quick, but now they are like a standard in data analysis and data science, with those people needing experienced devs to translate the notebook into production code. This just drives the silo walls up higher.

> I don't get why anyone one who knows how to use an IDE would ever use a notebook, The Python IDEs for data science are mostly garbage - if you have any recommendations, I'm all ears because I really don't like notebooks but still keep switching between jupyter and vscode depending on what I'm working on.

I use IntelliJ for all my work, data or normal dev stuff, and it works great (all is python). Maybe there is just a workflow issue here where people are used to saving their data as they go in cells. I just write my algorithms all the way through, get a subset of data to debug against, then use the debugger to help me see what mistakes I made. I always run my code all the way through and only stop at the step I'm debugging. I like this better than saving the data from previous computations because I tend to refactor a lot and would need to rerun most of the notebook anyway. Also, rerunning it all the way through a lot makes me notice slow spots more than if I only ran that area a few times and saved the results. For me, this has the effect that those areas get more attention and my code is closer to production grade than if I had used a notebook workflow. My two cents, but give IntelliJ a try if you want a good python IDE.

Re: What’s wrong with computational notebooks?

#172

Post by rjdagost »

vnpc1 wrote: post #110
comment_guy wrote: post #59

I don't get why anyone one who knows how to use an IDE would ever use a notebook, the coding experience is garbage in comparison. I understand they started as a way to get STEM kids coding quick, but now they are like a standard in data analysis and data science, with those people needing experienced devs to translate the notebook into production code. This just drives the silo walls up higher.

> I don't get why anyone one who knows how to use an IDE would ever use a notebook, The Python IDEs for data science are mostly garbage - if you have any recommendations, I'm all ears because I really don't like notebooks but still keep switching between jupyter and vscode depending on what I'm working on.

I have found PyCharm to offer a good trade-off between data exploration and productionizing your code. It has the best Python debugger that I've used. You can also run Jupyter notebooks in PyCharm when that makes sense for you.

Re: What’s wrong with computational notebooks?

#173

Post by comment_guy »

aldanor wrote: post #170
C1sc0cat wrote: post #167

Earlier quoted context omitted.

Maybe thinking about the data and what your trying to do before coding might be an idea as well.

'Thinking about the data' most often requires looking at the data from hundreds of different angles, quickly investigating its properties and statistics, maybe plotting or fitting a few things, checking some hypotheses etc (all of the above code you will most likely throw out after the initial stage). Same with the results - once you've coded something (perhaps outside of a notebook environment) and obtained results,…

Maybe you use a notebook I'm completely unfamiliar with, but my experience is that they allow you to write code, run it, and save the results in cells. My IDE does all of that except the saving of partial results part, but this can be done easily by just dumping your precomputed data to disk if you can't recompute it easily. In either case, an IDE gives you get an actual debugger, plus with IntelliJ it has a great data visualization plugins, database viewer, great autocompletion, and integrates with your VCS, etc. What do you do when you need an actual debugger, or need to profile your code? What about documentation for the function you are calling? In my IDE this is a popup, in every notebook I've used, this is a google search.

Re: What’s wrong with computational notebooks?

#174

Post by gdy »

playing_colours wrote: post #146
gdy wrote: post #107

Earlier quoted context omitted.

If you have a few long data loading and preprocessing steps it's a pain to wait for them to run again, people try to avoid it. When something odd begins to happen, they don't immediately consider the possibility that it's not their bug and waste time trying to 'debug' the problem instead of just rerunning the notebook.

Would it be a solution to store intermediate computations to an in-memory or disk database like Redis, SQLite? It is a matter of few minutes to run a docker instance and write simple read / write + serialize Python util functions?

Surely, it would be a solution, but I don't think for an average data scientist it's a matter of few minutes.

Re: What’s wrong with computational notebooks?

#175

Post by pjmlp »

ngcc_hk wrote: post #81
rb808 wrote: post #14

I tried to encourage our team to use notebooks, however everyone prefers using PyCharm and git for sharing code. We dont have much visualization, which might be the reason, but I was surprised just how many people just hated it.

Are you using oo? Still not sure how to “explain” an oo system once sophisticated enough. Just better than go-to everywhere but not much. Of course a trigger based system (gui, system) also have the same issue. This code + explanation would not work I guess.

> Still not sure how to “explain” an oo system once sophisticated enough.

With a couple of UML diagrams, still the best option.

Re: What’s wrong with computational notebooks?

#176

Post by teekert »

I don't know what this document is meant to do but you will have to take my Jupyter-lab instance from my dead cold hands.

I love notebooks, I work fast, line by line I execute commands and I immediately see the output (dataframes or graphs). For complex code I have an editor open (in jupyter-lab or vscode) for some functions and classes. But the main developing is done in the notebook, anything that ends in a module start in my notebooks.

As a biologist that learned to program after 30 I just don't understand how you can develop data processing code without such a close handle on dataframes and without checking in graphs/visualizations if your code does what you expect. I don't see how I would do that in pure vscode of other IDEs.

I also don't understand this sentence: "Once the data is loaded, it then has to be cleaned, which participants complained is a repetitive and time consuming task that involves copying and pasting code from their personal "library" of commonly used functions." What is the alternative? Not cleaning the code? And why copy and paste when you can perfectly fine have your own shareable module on the side? I guess most notebook users do some kind of hybrid development.

Re: What’s wrong with computational notebooks?

#177

Post by hprotagonist »

funklute wrote: post #140
hprotagonist wrote: post #2

See also Joel Grus' talk, "I Don't Like Notebooks": https://www.youtube.com/watch?v=7jiPeIFXb6U slides: https://docs.google.com/presentation/d/1n2RlMdmv1p25Xy5thJUh...

Superb talk! It's worth noting that a lot of the issues he brings up, ultimately stem from the format in which Jupyter notebooks are stored. R notebooks, with their plain-text stored format as well as code-chunk parameters, solve some, but not all, of these problems.

https://github.com/mwouts/jupytext

doesn’t solve the state management testing or tooling issues though, but commits are slightly less awful.

Re: What’s wrong with computational notebooks?

#178

Post by mbreese »

jpz wrote: post #117
snapetom wrote: post #42

Earlier quoted context omitted.

I just wanted to say thank you. Many of the points in your study strikes a nerve. Part of my responsibility at my last job was to introduce good software engineering practices. What happens? The data scientists go rogue and start running notebooks left and right. How do they productionize their work? Well, they don't. They were academics. All they know is that the models ran fine in their notebooks on their laptops.…

The tone of what you are saying strikes a nerve with me - we had exactly the same issues with Excel in the front office in investment banking. Unknowable ad-hoc, unversioned spreadsheets running much of the capital of the company.

That’s a really good comparison. Excel is often used for storing data and doing analysis because it just plain works. And anyone can use it.

Notebooks tend to be the same way. It’s a simple GUI-ish was to do many complex analyses in a quick and dirty way.

And many of the arguments for not using Excel are the same as not using notebooks. Each is good at the initial data exploration stage, but are often abused and used in production when everyone knows it is a bad idea. But it still “works” so it is unlikely to be replaced.

(Especially when those that are working with the data don’t always have the skill set to build out a full production workflow.)

Re: What’s wrong with computational notebooks?

#179

Post by archi42 »

PeterisP wrote: post #154
archi42 wrote: post #115

As a computer scientist/software engineer, please allow me the question: Why would I prefer a notebook over e.g. equivalent python script(s) in a git? I first saw jupyter notebooks when my sister (physicist, non-programmer) used it for analyzing economical data with pandas. Run-time for the full data set was half a day (and IMHO for that analysis SQL would have been better suited). I understand that as a non-programm…

The key factor is iteration speed. If step A takes 5 minutes (and 5 minutes is a very short time) and I want to experiment on step B, then I don't want to rerun step A each time while I'm writing and running code that helps me understand what step B is going to be; I'd want that to be interactive and immediate, not have each rerun take 5 minutes. Storing/loading to disk is not a good option because all the data that…

Ah, I think I have a hugely different approach to data processing: For my work I often have a very good idea what the output should look like, and what transformations are required on the input to get there. E.g. when processing log files to generate an overview page, or (as I'm doing right now) adding a target to binutils (assembler, linker,...). (Obviously I'm not a data scientist ;-)

With what you describe, intuitively I would use a library that allows me to store&load data per step (with verifying the structure matches), or pass it in-memory. Think JSON (yeah, slow) or something like protobuffers. That way I could do both

> store(A(read(input)) -> file); store(B(load(file)) -> file2)

during development (or in case B is in another language as A), and in production just

> B(A(read(some_other_input)))

But yeah, that's just my intuition of course. Maybe I'd be a bad data scientist.

However, can't you just experiment with smaller data sets? That's what I usually do if processing is slow (e.g. instead of parsing 10GB of log files, I'll just do 50MB to verify the processing pipeline works, and once that's it, run it on the full 10GB and grab a coffee while it runs). Not an option for data science?

Post reply on HN