Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Data Science at the Command Line

datascienceatthecommandline.com

Post reply on HN
Print view View original

31–36 of 36 posts

Re: Data Science at the Command Line

#31

Post by minimaxir »

IanCal wrote: post #29
minimaxir wrote: post #11

Earlier quoted context omitted.

Notebooks (and this command-line ebook) assume that the input data is static (i.e. an ad-hoc analysis) which is a more typical use case. Dynamic data/reporting is a different thing entirely, at which point things like business intelligence software and dashboards come into play, and outside the scope of a command line anyways.

Notebooks however let you run code blocks in arbitrary orders, delete ones but keep their results in memory, change them without rerunning and change code with none of the downstream dependencies updating. It's possible (actually very easy) to have code which works as you're making it but not if you run it from scratch.

Which is why notebooks typically a) declare all imports/dependencies in the first block and b) the entire notebook is rerun from a fresh session before publishing (and notebooks have a keyboard command for doing that too).

In all my years of work with Notebooks, I've never had an issue with downstream dependencies.

Re: Data Science at the Command Line

#32

Post by make3 »

goatlover wrote: post #30
make3 wrote: post #13

Earlier quoted context omitted.

notebooks are garbage because of the arbitrary order of execution

The cells have numbers corresponding to execution, and you can restart and rerun the entire notebook, or down to a certain cell very easily.

jupyter specifically discourage this by hiding the button to run the whole notebook in a menu and by not assigning a shortcut to the functionality. It's better in colab though, where at least there is a button.

The thing is that people who are not professional programmers often don't realize that this is a danger, and their thing doesn't work and they don't know that it's because they're in a weird state emanating from the random execution of code blocks. So, I mean, notebooks are useful and cool, but they're definitely dangerous, especially for people who aren't software engineers, which is likely a huge fraction of their users

Re: Data Science at the Command Line

#33

Post by achompas »

minimaxir wrote: post #11
achompas wrote: post #9

Earlier quoted context omitted.

Notebooks are not much better than copy-pasting from a notepad or editor into an interpreter. They’re great for reports, but dangerous for presenting the illusion of reproducibility. At best you’re constantly restarting your kernel and clearing output. More likely, output from cell #7 has modified output [138] but you haven’t updated the chart produced in cell #17 (or some similar craziness). Not much better than pro…

Notebooks (and this command-line ebook) assume that the input data is static (i.e. an ad-hoc analysis) which is a more typical use case. Dynamic data/reporting is a different thing entirely, at which point things like business intelligence software and dashboards come into play, and outside the scope of a command line anyways.

I might have needed to edit down my comment, but you locked on to the least important part of my argument.

If reproducibility is important, like you say, then a notebook is the last thing you want. Your code needs to be tested and designed like the software it is, instead of tossed into some notebook that does not fit into classic software practices.

Re: Data Science at the Command Line

#34

Post by achompas »

minimaxir wrote: post #31
IanCal wrote: post #29

Earlier quoted context omitted.

Notebooks however let you run code blocks in arbitrary orders, delete ones but keep their results in memory, change them without rerunning and change code with none of the downstream dependencies updating. It's possible (actually very easy) to have code which works as you're making it but not if you run it from scratch.

Which is why notebooks typically a) declare all imports/dependencies in the first block and b) the entire notebook is rerun from a fresh session before publishing (and notebooks have a keyboard command for doing that too). In all my years of work with Notebooks, I've never had an issue with downstream dependencies.

Important distinction: the notebooks themselves don't do this. The notebook authors, by convention, are supposed to do this.

This is no different than engineers who are, by convention, supposed to write bug-free code. Even with this convention, however, devs still rely on testing to decrease the likelihood of bugs.

No similar tooling exists for notebooks, which is why I recommend moving re-used logic into a tested codebase.

Re: Data Science at the Command Line

#35

Post by sedachv »

arca_vorago wrote: post #27
peatmoss wrote: post #14

Earlier quoted context omitted.

I very frequently do data tasks as bash command lines and do so within an org-mode code block. So, at least with org-mode, notebook computing and data processing in the shell are not mutually exclusive. EDIT: Also I should note that notebooks are not the only (or in my opinion best) way to present a reproducible analysis.

I'm working on my data science degree (previously a senior sysadmin), and this is also the workflow I have settled on. Along with versioning/diffs so you can walk back in time to see your changes in a script, I think it's one of the more robust and reproduceable systems around. So my scripts focus on the data, but around the code block are comments and notes about what is being done, etc. I've recently been looking a…

> If it werent for browsing the internet (eww in emacs is nice though) and gaming, I don't think I'd even need a desktop environment.

I switched to EXWM[1]; Firefox looks and acts like an Emacs buffer.

[1] https://github.com/ch11ng/exwm

Post reply on HN