Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Data Science at the Command Line

datascienceatthecommandline.com

Post reply on HN
Print view View original

11–20 of 36 posts

Re: Data Science at the Command Line

#11

Post by minimaxir »

achompas wrote: post #9
minimaxir wrote: post #4

While the efficiency of a command line is always sexy, but for data science in particular , where reproducibility is important and bugs are subtle and often don't cause a terminal error, it is worth it to sacrifice a little bit of code efficiency for code clarity in the long run by using an IDE/Notebook.

Notebooks are not much better than copy-pasting from a notepad or editor into an interpreter. They’re great for reports, but dangerous for presenting the illusion of reproducibility. At best you’re constantly restarting your kernel and clearing output. More likely, output from cell #7 has modified output [138] but you haven’t updated the chart produced in cell #17 (or some similar craziness). Not much better than pro…

Notebooks (and this command-line ebook) assume that the input data is static (i.e. an ad-hoc analysis) which is a more typical use case.

Dynamic data/reporting is a different thing entirely, at which point things like business intelligence software and dashboards come into play, and outside the scope of a command line anyways.

Re: Data Science at the Command Line

#12

Post by utefan001 »

jeroenjanssens wrote: post #3

Believe it or not, it's partly thanks to you, HN, that I wrote this book in 2014 in the first place [1]! It's humbling to see it listed here again now that the text has become available under a CC BY-ND license. Thank you for your help in spreading the word. All this attention (read: likes, shares, and page views) is making me wonder whether it's worthwhile to write an update (or even a second edition). What do you t…

sort | uniq -c | sort -n example in chapter 5 is something I use a lot. Do you typically use Mac or Linux? The way Mac does threading made sort 16 times slower on 1GB+ size files compared to my $300 old Lenovo laptop running Ubuntu.

Re: Data Science at the Command Line

#13

Post by make3 »

minimaxir wrote: post #4

While the efficiency of a command line is always sexy, but for data science in particular , where reproducibility is important and bugs are subtle and often don't cause a terminal error, it is worth it to sacrifice a little bit of code efficiency for code clarity in the long run by using an IDE/Notebook.

notebooks are garbage because of the arbitrary order of execution

Re: Data Science at the Command Line

#14

Post by peatmoss »

minimaxir wrote: post #4

While the efficiency of a command line is always sexy, but for data science in particular , where reproducibility is important and bugs are subtle and often don't cause a terminal error, it is worth it to sacrifice a little bit of code efficiency for code clarity in the long run by using an IDE/Notebook.

I very frequently do data tasks as bash command lines and do so within an org-mode code block. So, at least with org-mode, notebook computing and data processing in the shell are not mutually exclusive.

EDIT: Also I should note that notebooks are not the only (or in my opinion best) way to present a reproducible analysis.

Re: Data Science at the Command Line

#17

Post by fnl »

Given the number of tools around, and as this book promotes Drake, has anyone got "comparative experience" with some of the following tools:

Cookiecutter: https://drivendata.github.io/cookiecutter-data-science

DataVersionControl: https://dataversioncontrol.com/

Drake: https://github.com/Factual/drake

Luigi: https://github.com/spotify/luigi

Pachyderm: http://www.pachyderm.io/

Sacred: https://github.com/IDSIA/sacred

They all focus in slightly different ways on the issue of managing data science/machine learning workflows, so I wonder if someone has a clear preference for one of those over any another.

EDIT: added Luigi

Re: Data Science at the Command Line

#18

Post by fnl »

fnl wrote: post #17

Given the number of tools around, and as this book promotes Drake, has anyone got "comparative experience" with some of the following tools: Cookiecutter: https://drivendata.github.io/cookiecutter-data-science DataVersionControl: https://dataversioncontrol.com/ Drake: https://github.com/Factual/drake Luigi: https://github.com/spotify/luigi Pachyderm: http://www.pachyderm.io/ Sacred: https://github.com/IDSIA/sacred Th…

To amend, why I'm even bringing this up, what worries me about Drake is this: https://github.com/Factual/drake/pulse/monthly Its GitHub pulse is - dead; For two years now. Makes me think one of the other projects listed might be better choices.

Re: Data Science at the Command Line

#19

Post by vondur »

utefan001 wrote: post #12
jeroenjanssens wrote: post #3

Believe it or not, it's partly thanks to you, HN, that I wrote this book in 2014 in the first place [1]! It's humbling to see it listed here again now that the text has become available under a CC BY-ND license. Thank you for your help in spreading the word. All this attention (read: likes, shares, and page views) is making me wonder whether it's worthwhile to write an update (or even a second edition). What do you t…

sort | uniq -c | sort -n example in chapter 5 is something I use a lot. Do you typically use Mac or Linux? The way Mac does threading made sort 16 times slower on 1GB+ size files compared to my $300 old Lenovo laptop running Ubuntu.

Have you tried installing it from homebrew? brew install coreutils This assumes you have brew installed.

Re: Data Science at the Command Line

#20

Post by swhalen »

minimaxir wrote: post #4

While the efficiency of a command line is always sexy, but for data science in particular , where reproducibility is important and bugs are subtle and often don't cause a terminal error, it is worth it to sacrifice a little bit of code efficiency for code clarity in the long run by using an IDE/Notebook.

It seems to me that encapsulating analyses in bash scripts would help with reproducibility.
Post reply on HN