Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Data Science at the Command Line

datascienceatthecommandline.com

Post reply on HN
Print view View original

21–30 of 36 posts

Re: Data Science at the Command Line

#21

Post by bringtheaction »

fnl wrote: post #18
fnl wrote: post #17

Given the number of tools around, and as this book promotes Drake, has anyone got "comparative experience" with some of the following tools: Cookiecutter: https://drivendata.github.io/cookiecutter-data-science DataVersionControl: https://dataversioncontrol.com/ Drake: https://github.com/Factual/drake Luigi: https://github.com/spotify/luigi Pachyderm: http://www.pachyderm.io/ Sacred: https://github.com/IDSIA/sacred Th…

To amend, why I'm even bringing this up, what worries me about Drake is this: https://github.com/Factual/drake/pulse/monthly Its GitHub pulse is - dead; For two years now. Makes me think one of the other projects listed might be better choices.

I was about to say maybe it's finished but it has 70 open issues so maybe it was just abandoned?

It is at version 1.0.3 though so it could be that it's considered finished. Seems strange to leave the issues open if it was though.

Re: Data Science at the Command Line

#22

Post by tejtm »

swhalen wrote: post #20
minimaxir wrote: post #4

While the efficiency of a command line is always sexy, but for data science in particular , where reproducibility is important and bugs are subtle and often don't cause a terminal error, it is worth it to sacrifice a little bit of code efficiency for code clarity in the long run by using an IDE/Notebook.

It seems to me that encapsulating analyses in bash scripts would help with reproducibility.

scripts marshalled with makefiles have helped me with long period recurrent tasks that also slowly evolve

Re: Data Science at the Command Line

#23

Post by utefan001 »

vondur wrote: post #19
utefan001 wrote: post #12

Earlier quoted context omitted.

sort | uniq -c | sort -n example in chapter 5 is something I use a lot. Do you typically use Mac or Linux? The way Mac does threading made sort 16 times slower on 1GB+ size files compared to my $300 old Lenovo laptop running Ubuntu.

Have you tried installing it from homebrew? brew install coreutils This assumes you have brew installed.

Yea, same issue with gnu coreutils. Mac uses a fairness threading algorithm, causing lots of time to be wasted deciding whose turn it is.

https://stackoverflow.com/questions/28888719/multi-threaded-...

Re: Data Science at the Command Line

#24

Post by bobivl »

I also used bash scripts a lot to get quick insights from csv files. Someday I realized that these are mostly sql queries that I encoded into complex scripts. For the sake of trying, I implemented a simple sql to bash transpiler that takes a sql query and returns a bash one-liner that you can execute on csv file(s).

Give it a try: http://bigbash.it

Re: Data Science at the Command Line

#25

Post by confounded »

minimaxir wrote: post #4

While the efficiency of a command line is always sexy, but for data science in particular , where reproducibility is important and bugs are subtle and often don't cause a terminal error, it is worth it to sacrifice a little bit of code efficiency for code clarity in the long run by using an IDE/Notebook.

> ... using an IDE/Notebook.

I was with you until the end. The important part is that a script is clear and commented, and can be written to fail informatively. These are benefits of using a scripting language.

Who cares if an IDE was used rather than a traditional editor?

Dynamic notebooks and the JSON mess they generate are a personal peeve of mine, and IME, the enemy of reproducibility.

If it’s for a one-off analysis, but processing is complicated enough that scripts don’t make dependencies between processing steps clear, use GNU Make.

If it’s a data product that’s running in the background, consider something like Airflow.

Still, I find coreutils and friends incredibly useful for interactively sizing up text data.

Re: Data Science at the Command Line

#26

Post by pjmorris »

minimaxir wrote: post #4

While the efficiency of a command line is always sexy, but for data science in particular , where reproducibility is important and bugs are subtle and often don't cause a terminal error, it is worth it to sacrifice a little bit of code efficiency for code clarity in the long run by using an IDE/Notebook.

Code clarity includes the interfaces of the tool of choice. I suspect that shell files checked in to a version control system will have a longer working life because IDE/Notebook interfaces are more likely to change and become incompatible over time driven by the commercial needs of their vendors.

Re: Data Science at the Command Line

#27

Post by arca_vorago »

peatmoss wrote: post #14
minimaxir wrote: post #4

While the efficiency of a command line is always sexy, but for data science in particular , where reproducibility is important and bugs are subtle and often don't cause a terminal error, it is worth it to sacrifice a little bit of code efficiency for code clarity in the long run by using an IDE/Notebook.

I very frequently do data tasks as bash command lines and do so within an org-mode code block. So, at least with org-mode, notebook computing and data processing in the shell are not mutually exclusive. EDIT: Also I should note that notebooks are not the only (or in my opinion best) way to present a reproducible analysis.

I'm working on my data science degree (previously a senior sysadmin), and this is also the workflow I have settled on. Along with versioning/diffs so you can walk back in time to see your changes in a script, I think it's one of the more robust and reproduceable systems around.

So my scripts focus on the data, but around the code block are comments and notes about what is being done, etc.

I've recently been looking at how to integrate makefile into this since watching the following video: https://www.youtube.com/watch?v=sd0HhW8vkSQ

All that said, kudos to the author for this, reading through it I see a few things I hadn't even though of yet, and I live on the command line in general. If it werent for browsing the internet (eww in emacs is nice though) and gaming, I don't think I'd even need a desktop environment.

Re: Data Science at the Command Line

#28

Post by fnl »

bringtheaction wrote: post #21
fnl wrote: post #18

Earlier quoted context omitted.

To amend, why I'm even bringing this up, what worries me about Drake is this: https://github.com/Factual/drake/pulse/monthly Its GitHub pulse is - dead; For two years now. Makes me think one of the other projects listed might be better choices.

I was about to say maybe it's finished but it has 70 open issues so maybe it was just abandoned? It is at version 1.0.3 though so it could be that it's considered finished. Seems strange to leave the issues open if it was though.

Maybe so. Yet, I can't find any features in Drake that I don't get with Make, too - in fact, it looks to be rather the opposite.

Indeed, for some of the tools I listed, they barely have any more functionality than I'd get out of Make & Git alone. And for Make, I'm pretty sure development & support will stick around for a few more years...

To me, only Luigi (Hadoop integration), Pachyderm (containers, production deployments in the Enterprise version) and Sacred (Python & TensorFlow integration) really stick out as differentiating themselves. But maybe I'm overlooking something?

Re: Data Science at the Command Line

#29

Post by IanCal »

minimaxir wrote: post #11
achompas wrote: post #9

Earlier quoted context omitted.

Notebooks are not much better than copy-pasting from a notepad or editor into an interpreter. They’re great for reports, but dangerous for presenting the illusion of reproducibility. At best you’re constantly restarting your kernel and clearing output. More likely, output from cell #7 has modified output [138] but you haven’t updated the chart produced in cell #17 (or some similar craziness). Not much better than pro…

Notebooks (and this command-line ebook) assume that the input data is static (i.e. an ad-hoc analysis) which is a more typical use case. Dynamic data/reporting is a different thing entirely, at which point things like business intelligence software and dashboards come into play, and outside the scope of a command line anyways.

Notebooks however let you run code blocks in arbitrary orders, delete ones but keep their results in memory, change them without rerunning and change code with none of the downstream dependencies updating.

It's possible (actually very easy) to have code which works as you're making it but not if you run it from scratch.

Re: Data Science at the Command Line

#30

Post by goatlover »

make3 wrote: post #13
minimaxir wrote: post #4

While the efficiency of a command line is always sexy, but for data science in particular , where reproducibility is important and bugs are subtle and often don't cause a terminal error, it is worth it to sacrifice a little bit of code efficiency for code clarity in the long run by using an IDE/Notebook.

notebooks are garbage because of the arbitrary order of execution

The cells have numbers corresponding to execution, and you can restart and rerun the entire notebook, or down to a certain cell very easily.
Post reply on HN