Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: As a data scientist, what should be in my toolkit in 2018?

news.ycombinator.com

Post reply on HN
Print view View original

71–80 of 177 posts

Re: Ask HN: As a data scientist, what should be in my toolkit in 2018?

#71

Post by red_hare »

fsloth wrote: post #62
ms013 wrote: post #22

Mathematics. Which branch of math is domain dependent. Stats come up everywhere. Graphs do too. In addition to baseline math, you really need to understand the problem domain and goals of the analysis. Languages and libraries are just tools: knowing APIs doesn’t tell you at all how to solve a problem. They just give you things to throw at a problem. You need to know a few tools, but to be honest, they’re easy and you…

Do people get careers as 'data scientists' without masters degrees? I'm heavily biases towards experts... but still would call it fair in the general case to call anyone doing data science without at least a masters degree or the equivalent mental toolkit more like a 'data quack'.

I agree that a graduate degree goes a long way. Even though the requirements of the role varies greatly from team to team, it almost always involves going out to learn and apply new math to solve a problem. That's one of those things a graduate degree (humanities as well, not just STEM) tends to require of you. And a Phd demonstrates you've done that for years.

Re: Ask HN: As a data scientist, what should be in my toolkit in 2018?

#72

Post by barskern »

p33p wrote: post #56
ms013 wrote: post #22

Mathematics. Which branch of math is domain dependent. Stats come up everywhere. Graphs do too. In addition to baseline math, you really need to understand the problem domain and goals of the analysis. Languages and libraries are just tools: knowing APIs doesn’t tell you at all how to solve a problem. They just give you things to throw at a problem. You need to know a few tools, but to be honest, they’re easy and you…

And don’t over use viz. Pictures do effectively communicate, but often people visualize without understanding. The result is pretty pictures that eventually people realize communicate little effective domain insight. You’d be surprised that sometimes simple and ugly pictures communicate more insight than beautiful ones do. I don't necessarily agree with this. Yes, a sound understanding of the domain and knowledge of…

> And don’t over use viz. [...]

I don't fully agree with this neither. Especially for mathematical concepts, visualization can give insight into how theorems are constructed and combined. This can prove to be vital when applying concepts and theorems to new problems.

I would especially like to bring forth 3blue1brown[1]. He is a creator of videos which beautifully visualizes and explains complex mathematical problems. His efforts has given me an insight into math which theorems explained in text and variables could never do.

However I do see your point that visualizations without understanding can be misleading. Hence the pure, written math is important to read and reason about, but I do believe that some concepts need to be visualized to be fully understood.

[1]: http://www.3blue1brown.com

Re: Ask HN: As a data scientist, what should be in my toolkit in 2018?

#73

Post by throwaway7645 »

Aardwolf wrote: post #67
ms013 wrote: post #22

Mathematics. Which branch of math is domain dependent. Stats come up everywhere. Graphs do too. In addition to baseline math, you really need to understand the problem domain and goals of the analysis. Languages and libraries are just tools: knowing APIs doesn’t tell you at all how to solve a problem. They just give you things to throw at a problem. You need to know a few tools, but to be honest, they’re easy and you…

R is not present in your list, did you ever try it and what's your opinion about it?

If you have Mathematica, you might not need R as both are like Swiss Army Chainsaws for Data Analytics.

Re: Ask HN: As a data scientist, what should be in my toolkit in 2018?

#74

Post by elsherbini »

I'm a scientist (PhD student in microbiolgy) that works with lots of data. My data is on the order of hundreds of gigabytes (genome collections and other sequencing data) or megabytes (flat files).

I use the `tidyverse` from R[0] for everything people use `pandas` for. I think the syntax is soooo much more pleasant to use. It's declarative and because of pipes and "quosures" is highly readable. Combined with the power of `broom`,fitting simple models to the data and working with the results is really nice. Add to that that `ggplot` (+ any sane styling defaults like `cowplot`) is the fastest way to iterate on data visualizations that I've ever found. "R for Data Science" [1] is great free resource for getting started.

Snakemake [2] is a pipeline tool that submits steps of the pipeline to a cluster and handles waiting for steps to finish before submitting dependent steps. As a result, my pipelines have very little boilerplate, they are self documented, and the cluster is abstracted away so the same pipeline can work on a cluster or a laptop.

[0] https://www.tidyverse.org/

[1] http://r4ds.had.co.nz/

[2] http://snakemake.readthedocs.io/en/stable/

Re: Ask HN: As a data scientist, what should be in my toolkit in 2018?

#75

Post by cirgue »

fsloth wrote: post #62
ms013 wrote: post #22

Mathematics. Which branch of math is domain dependent. Stats come up everywhere. Graphs do too. In addition to baseline math, you really need to understand the problem domain and goals of the analysis. Languages and libraries are just tools: knowing APIs doesn’t tell you at all how to solve a problem. They just give you things to throw at a problem. You need to know a few tools, but to be honest, they’re easy and you…

Do people get careers as 'data scientists' without masters degrees? I'm heavily biases towards experts... but still would call it fair in the general case to call anyone doing data science without at least a masters degree or the equivalent mental toolkit more like a 'data quack'.

Yes, they do. Having a masters or not is orthogonal to being a quack, especially in an environment as buzzword-laden as data science/ML.

Re: Ask HN: As a data scientist, what should be in my toolkit in 2018?

#76

Post by wenc »

anc84 wrote: post #14

Any programming language that you are proficient in. A solid understanding how a computer works. Solid basis of statistics. Anything else is just sprinkles, trends and field-specific.

> Any programming language that you are proficient in.

Oh I don't know about that. Programming languages are force multipliers, and each language has a different force coefficients for different problem domains. They are not all equivalent. They have their different points of leverage, and simply being good in one does not mean you can solve problems in any domain with ease. In fact the wrong programming language can often be harmful if it's ill-suited to the problem at hand, and especially if it contorts your mental model of what you can do with the data.

One example I encounter a lot in industry is Excel VBA. I'm fairly good at VBA and have seen very sophisticated code in VBA. I've also seen many basic operations implemented badly in VBA that should not have been written in VBA at all. By solving the problem in VBA, the solution is often "hemmed in" by the constraints of VBA.

For instance, unpivoting data is often done badly in VBA (with for-loops), but is trivial to do well in dplyr or pandas.

So I would say one has to choose one's programming language somewhat carefully. Not any language will do.

Re: Ask HN: As a data scientist, what should be in my toolkit in 2018?

#79

Post by xitrium »

If you care about quantifying uncertainty, knowing about Bayesian methods is a good idea I don't see represented here yet. I care so much about uncertainty quantification and propagation that I work on the Stan project[0] which has an extremely complete manual (600+ pages) and many case studies illustrating different problems. Full Bayesian inference such as that provided by Stan's Hamiltonian Monte Carlo inference algorithm is fairly computationally expensive so if you have more data than fits into RAM on a large server, you might be better served by some approximate methods (but note the required assumptions) like INLA[1].

[0] http://mc-stan.org/ [1] http://www.r-inla.org/

Re: Ask HN: As a data scientist, what should be in my toolkit in 2018?

#80

Post by nonbel »

elsherbini wrote: post #74

I'm a scientist (PhD student in microbiolgy) that works with lots of data. My data is on the order of hundreds of gigabytes (genome collections and other sequencing data) or megabytes (flat files). I use the `tidyverse` from R[0] for everything people use `pandas` for. I think the syntax is soooo much more pleasant to use. It's declarative and because of pipes and "quosures" is highly readable. Combined with the powe…

Sometimes I think I'm the only one who isn't really a fan of the tidyverse. I've found it slower, more prone to dependency issues, more prone to silent errors, and less well documented than most R packages (ie most of what you find on CRAN).
Post reply on HN