Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: As a data scientist, what should be in my toolkit in 2018?

news.ycombinator.com

Post reply on HN
Print view View original

131–140 of 177 posts

Re: Ask HN: As a data scientist, what should be in my toolkit in 2018?

#131

Post by Balgair »

ms013 wrote: post #22

Mathematics. Which branch of math is domain dependent. Stats come up everywhere. Graphs do too. In addition to baseline math, you really need to understand the problem domain and goals of the analysis. Languages and libraries are just tools: knowing APIs doesn’t tell you at all how to solve a problem. They just give you things to throw at a problem. You need to know a few tools, but to be honest, they’re easy and you…

Honestly, in any STEM major, esp. the physics heavy ones, those maths areas should be well understood. Is any (physics heavy) STEM major also a Data Scientist then too?

Re: Ask HN: As a data scientist, what should be in my toolkit in 2018?

#132

Post by alexbeloi »

dsacco wrote: post #128
alexbeloi wrote: post #125

Earlier quoted context omitted.

I would amend as follows: Skip abstract algebra, topology and analysis. If you find yourself in the same room as a number theory book, walk away slowly without making eye contact lest it cast a spell on you.

If you skip analysis and at least elementary topology your understanding will be limited to discrete probability, at best. If you skip abstract algebra, you’ll miss out on a lot of buildup to advanced linear transformations and operations in vector spaces. Sure, skip number theory. Like I said, you could swap that out.

You could also spend months of your life learning the intricacies of measure theory, but then every measure in probability is a positive sigma-additive unit measure.

One can learn the necessary topology, analysis (etc.) in the relevant places (and the relevant depths) that they come up.

Re: Ask HN: As a data scientist, what should be in my toolkit in 2018?

#133

Post by sandGorgon »

wenc wrote: post #66
sandGorgon wrote: post #9

Earlier quoted context omitted.

same question that i have. Anyone using pyspark in production ? Would you use pyspark mllib in a webservice instead of scikit ?

1) Yes, PySpark is great if you're mostly just doing dataframe manipulation in Spark, using built-in functions. PySpark actually has similar performance to Scala Spark for dataframes. (We've moved away from RDDs) However, if you use a lot of UDFs where Spark has to serialize your Python functions, you might consider rewriting those UDFs in a JVM language. Serialization overhead is still fairly substantial. Arrow is t…

Hi, Thanks for the answer. What you said resonates with me - with a few changes. Spark 2.3 will come with Arrow UDF, that should be a significant performance boost. In that way, yes - we are taking at a forward looking bet.

About mllib - yes, we concur with you on algorithmic coverage. And yes, training is the major issue. For example, what I read of Uber's Michaelangelo infrastructure - it seems they train using Spark and save to a custom format that is deserialized (using custom code) and made available as a docker image .

There is value in consistency - using Spark thtoy2and through. Wonder what you thought of that ?

Re: Ask HN: As a data scientist, what should be in my toolkit in 2018?

#135

Post by DataWorker »

nonbel wrote: post #80
elsherbini wrote: post #74

I'm a scientist (PhD student in microbiolgy) that works with lots of data. My data is on the order of hundreds of gigabytes (genome collections and other sequencing data) or megabytes (flat files). I use the `tidyverse` from R[0] for everything people use `pandas` for. I think the syntax is soooo much more pleasant to use. It's declarative and because of pipes and "quosures" is highly readable. Combined with the powe…

Sometimes I think I'm the only one who isn't really a fan of the tidyverse. I've found it slower, more prone to dependency issues, more prone to silent errors, and less well documented than most R packages (ie most of what you find on CRAN).

You are not alone. I think it’s a great thing for some people, but a net negative for the R community in the long run.

Re: Ask HN: As a data scientist, what should be in my toolkit in 2018?

#136

Post by throwaway713 »

minimaxir wrote: post #103
fsloth wrote: post #62

Earlier quoted context omitted.

Do people get careers as 'data scientists' without masters degrees? I'm heavily biases towards experts... but still would call it fair in the general case to call anyone doing data science without at least a masters degree or the equivalent mental toolkit more like a 'data quack'.

During my recent data science job hunt, I received a lot of resistance due to my lack of a Master's/PhD: https://twitter.com/minimaxir/status/951117788835278848

I read your tweets and totally agree that data science interviewing is really bizarre. I have a PhD and currently work as a principal data scientist at a large company, but I've been interviewing with some other companies and the interviews are outright strange.

I've had questions ranging from reversing strings on a whiteboard to checking for valid email addresses. I had another question about flipping biased coins and calculating probabilities. It's all nonsense and totally unrelated to the skills I developed during my PhD which primarily consisted of performing massive amounts of machine learning on high performance computing systems over large sets of data to extract important insights.

But — if solving these algorithm puzzles quickly and without errors is the key to a $300k+ job, so be it. I'll just practice this nonsense until I've optimized for the skill of "interviewing", and then maybe I can contribute in some kind of meaningful way to the company with actual data science.

Re: Ask HN: As a data scientist, what should be in my toolkit in 2018?

#137

Post by vobios »

nonbel wrote: post #80
elsherbini wrote: post #74

I'm a scientist (PhD student in microbiolgy) that works with lots of data. My data is on the order of hundreds of gigabytes (genome collections and other sequencing data) or megabytes (flat files). I use the `tidyverse` from R[0] for everything people use `pandas` for. I think the syntax is soooo much more pleasant to use. It's declarative and because of pipes and "quosures" is highly readable. Combined with the powe…

Sometimes I think I'm the only one who isn't really a fan of the tidyverse. I've found it slower, more prone to dependency issues, more prone to silent errors, and less well documented than most R packages (ie most of what you find on CRAN).

> less well documented than most R packages

I on the other hand, find most R packages provide barely readable documentation. I can just hope that the vignette exists and actually explains the inputs/outputs.

Re: Ask HN: As a data scientist, what should be in my toolkit in 2018?

#138

Post by vobios »

DataWorker wrote: post #135
nonbel wrote: post #80

Earlier quoted context omitted.

Sometimes I think I'm the only one who isn't really a fan of the tidyverse. I've found it slower, more prone to dependency issues, more prone to silent errors, and less well documented than most R packages (ie most of what you find on CRAN).

You are not alone. I think it’s a great thing for some people, but a net negative for the R community in the long run.

A non-insignificant fraction of the R community only exists because of Tidyverse.

Re: Ask HN: As a data scientist, what should be in my toolkit in 2018?

#139

Post by vobios »

Seanny123 wrote: post #115
chasedehan wrote: post #94

Earlier quoted context omitted.

I really enjoy the tidyverse, especially dplyr. I do most of my work in python now and find myself moving more and more of time in python. There are definitely some issues if you have to reliably run scripts (not to mention the difficulties of putting into production) The thing I really like about R over python is for SPECIFIC tasks like inspecting data and trying to get an answer out quickly, there really isn't a qu…

I would love some examples where R makes it easier to get answers than Python. Probably would make good Pandas2 issue too!

This is super controversial, but no need to use Jupyter.

I personally find that Jupyter feels like a hack compared to something like RStudio. You have to open a terminal and launch a web server?

Re: Ask HN: As a data scientist, what should be in my toolkit in 2018?

#140

Post by hobolord »

xitrium wrote: post #79

If you care about quantifying uncertainty, knowing about Bayesian methods is a good idea I don't see represented here yet. I care so much about uncertainty quantification and propagation that I work on the Stan project[0] which has an extremely complete manual (600+ pages) and many case studies illustrating different problems. Full Bayesian inference such as that provided by Stan's Hamiltonian Monte Carlo inference a…

do you have a recommended guide/textbook on learning stan? I've recently started doing more bayesian analysis, mainly bayesian estimation supercedes the t-test.
Post reply on HN