Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What’s wrong with computational notebooks?

web.eecs.utk.edu

Post reply on HN
Print view View original

81–90 of 223 posts

Re: What’s wrong with computational notebooks?

#81

Post by ngcc_hk »

rb808 wrote: post #14

I tried to encourage our team to use notebooks, however everyone prefers using PyCharm and git for sharing code. We dont have much visualization, which might be the reason, but I was surprised just how many people just hated it.

Are you using oo? Still not sure how to “explain” an oo system once sophisticated enough. Just better than go-to everywhere but not much. Of course a trigger based system (gui, system) also have the same issue.

This code + explanation would not work I guess.

Re: What’s wrong with computational notebooks?

#82

Post by MiroF »

ssivark wrote: post #71
snapetom wrote: post #42

Earlier quoted context omitted.

I just wanted to say thank you. Many of the points in your study strikes a nerve. Part of my responsibility at my last job was to introduce good software engineering practices. What happens? The data scientists go rogue and start running notebooks left and right. How do they productionize their work? Well, they don't. They were academics. All they know is that the models ran fine in their notebooks on their laptops.…

I just happened to be reading Peter Naur's "Programming as theory building" recently. It strikes me that taking its theme even a little seriously helps understand why notebooks are so popular. Notebooks happen to be convenient tools for exploring a new domain (interactively). Irrespective of how much software purists might complain, conventional software engineering provides very few tools/solutions/practices for tha…

I just refactor into a module that I import into my notebook as I go along. This lets me use the notebook for quick prototyping, but also productionize faster if need be.

Re: What’s wrong with computational notebooks?

#83

Post by anchpop »

anonsivalley652 wrote: post #80

There are these and other problems with CNs: 0. They try to be "be-all, end-all" proprietary container documents, so they lack generality, compatibility and embeddability. It would be better if live code try-out snippets were self-contained and embeddable in other documents: HTML, other software, maybe PDF, LaTex or literate programming formats. Maybe there should be standard, versioned interpreters for each kind of…

> Maybe there should be standard, versioned interpreters for each kind of programming language in WebAssembly and cached for offline usage by the browser for inclusion in documentation, papers, etc.

This would be incredible. Even better, the output from the code (like graphs) should be able to be embedded in the paper. You have no idea how many papers have errors in the code that generated the graphs/statistics/etc. and nobody can tell because the authors rarely release the data, let alone the source

Re: What’s wrong with computational notebooks?

#84

Post by weiser »

azhenley wrote: post #10

Co-author of the study here. Let me know if you have any questions or how you overcome some of the problems we identified!

In my experience, the best approach is to treat the notebook as the frontend. So widgets, graphs, annotations are generally ok. Anything compute intensive should be relegated to the backend.

Re: What’s wrong with computational notebooks?

#85

Post by anonsivalley652 »

anchpop wrote: post #83
anonsivalley652 wrote: post #80

There are these and other problems with CNs: 0. They try to be "be-all, end-all" proprietary container documents, so they lack generality, compatibility and embeddability. It would be better if live code try-out snippets were self-contained and embeddable in other documents: HTML, other software, maybe PDF, LaTex or literate programming formats. Maybe there should be standard, versioned interpreters for each kind of…

> Maybe there should be standard, versioned interpreters for each kind of programming language in WebAssembly and cached for offline usage by the browser for inclusion in documentation, papers, etc. This would be incredible. Even better, the output from the code (like graphs) should be able to be embedded in the paper. You have no idea how many papers have errors in the code that generated the graphs/statistics/etc.…

For WASM, there ought to be a package-management/registry mechanism for installation (unless there is already? It might get complicated, but would seem a good idea to reuse code/plugins.)... or as below, there ought to be some caching priority mechanism.

Then for HTML assets (and CSS ones too), perhaps a hint on asset-linking tags (a, script, link, img, audio, video, etc.) there ought to an offline-priority attribute to help the browser decide what to throw away when clearing cache the regular way or evicting items from the cache, while being able to leave some things deemed vital when not nuking the entire cache. Yes, websites could be goofy and game caching mechanisms, marking everything "vital" like for 0-pixel image cookies but I'm sure someone would make an "RBL" (real-time blackhole list) system of which priorities on which websites to ignore.

Related aside: There's a lot of common frameworks, libraries and bits that could be cached user-side, with the trick either to a) herding web devs to de-fragment their CDNs, which could create SPoF's or b) changing the standard allowing multiple SRCs or HREFs for high-availability/less bitrot to preserve both choice and encourage de-duplication of common assets. [0]

0. https://html.spec.whatwg.org/multipage/links.html#attr-hyper...

Re: What’s wrong with computational notebooks?

#87

Post by bllguo »

grp000 wrote: post #69
comment_guy wrote: post #59

I don't get why anyone one who knows how to use an IDE would ever use a notebook, the coding experience is garbage in comparison. I understand they started as a way to get STEM kids coding quick, but now they are like a standard in data analysis and data science, with those people needing experienced devs to translate the notebook into production code. This just drives the silo walls up higher.

Doing data science in an IDE would be terrible. With a notebook, you get the chance to load the data, view it, clean it where needed, view it again, analyze it, model it and do anything else you need to it. An IDE means that you can't use the previous output to guide your next operation in a direct fashion like you can with a notebook.

as a counterpoint, plenty of R folks are pretty happy doing all of that in Rstudio

Re: What’s wrong with computational notebooks?

#90

Post by kyllo »

bllguo wrote: post #87
grp000 wrote: post #69

Earlier quoted context omitted.

Doing data science in an IDE would be terrible. With a notebook, you get the chance to load the data, view it, clean it where needed, view it again, analyze it, model it and do anything else you need to it. An IDE means that you can't use the previous output to guide your next operation in a direct fashion like you can with a notebook.

as a counterpoint, plenty of R folks are pretty happy doing all of that in Rstudio

I'm an R folk, and I'm even happy doing all of that in Emacs!
Post reply on HN