Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: As a data scientist, what should be in my toolkit in 2018?

news.ycombinator.com

Post reply on HN
Print view View original

141–150 of 177 posts

Re: Ask HN: As a data scientist, what should be in my toolkit in 2018?

#141

Post by Xcelerate »

red_hare wrote: post #69
booleandilemma wrote: post #64

Earlier quoted context omitted.

I think data scientist, much like software engineer, is something you can call yourself without having any credentials whatsoever. It’s why technical interviews can be so brutal, unfortunately. There are a lot of frauds out there. Money attracts frauds. What’s the fizzbuzz test for data scientists anyway?

I'm a data engineer for a startup that's trying to hire its first data scientist. The range of candidates that apply with this title is massive. Defining our expectations has been challenging. My phone screen "fizzbuzz" is having them calculate a standard deviation from an array of data w/out with only basic operators (no numpy.std). Then explain why they choose population/sample and explain the difference. I studied…

> My phone screen "fizzbuzz" is having them calculate a standard deviation from an array of data w/out with only basic operators

This is just my n=1 opinion, but this is a terrible test for data science skills. I've had to calculate standard deviation by hand many times in my life, but my short term memory is such that despite doing that dozens of times over the past two decades, I still can't recall the formula off the top of my head. And then there's the whole n vs (n-1) thing in the denominator which has something to do with degrees of freedom, but I would just Google that as soon as I needed to know (depending on exactly what I was trying to do with the data).

So I don't understand how your question in any way tests someone's skills at analyzing data to extract valuable business insights. At best, it tests someone's ability to memorize formulas and minutiae (although I'll grant you that understanding the difference between a sample and the population is important).

Personally, I think take-home interviews with real data sets are the best way to gauge a candidate's skills. You're actually testing them with a work sample, and they are not under artificial time or memorization constraints.

Re: Ask HN: As a data scientist, what should be in my toolkit in 2018?

#142

Post by Xcelerate »

shepardrtc wrote: post #91
jerednel wrote: post #86

Earlier quoted context omitted.

At least with fizzbuzz you are working through how to logically solve a problem. This is just regurgitating a formula. I don't see how this is helpful.

Its designed to quickly weed out people who don't know the underlying math, just as FizzBuzz is designed to quickly weed out people who don't know programming.

I work as a data scientist, and my graduate research involved harmonic analysis over compact groups, optimization over Riemannian manifolds, and loopy belief propagation. You'd reject me in an interview because I couldn't remember the formula for standard deviation off the top of my head?

Re: Ask HN: As a data scientist, what should be in my toolkit in 2018?

#144

Post by fhk »

ms013 wrote: post #22

Mathematics. Which branch of math is domain dependent. Stats come up everywhere. Graphs do too. In addition to baseline math, you really need to understand the problem domain and goals of the analysis. Languages and libraries are just tools: knowing APIs doesn’t tell you at all how to solve a problem. They just give you things to throw at a problem. You need to know a few tools, but to be honest, they’re easy and you…

@ms013 interested to know how you are using the solvers, are you willing to share any further details?

Re: Ask HN: As a data scientist, what should be in my toolkit in 2018?

#145

Post by maayank »

ms013 wrote: post #22

Mathematics. Which branch of math is domain dependent. Stats come up everywhere. Graphs do too. In addition to baseline math, you really need to understand the problem domain and goals of the analysis. Languages and libraries are just tools: knowing APIs doesn’t tell you at all how to solve a problem. They just give you things to throw at a problem. You need to know a few tools, but to be honest, they’re easy and you…

I have good background in graph theory (IMHO) but don't know many data science use-cases (I'm amateur at that). Could you point to some good start points?

Re: Ask HN: As a data scientist, what should be in my toolkit in 2018?

#146

Post by wenc »

sandGorgon wrote: post #133
wenc wrote: post #66

Earlier quoted context omitted.

1) Yes, PySpark is great if you're mostly just doing dataframe manipulation in Spark, using built-in functions. PySpark actually has similar performance to Scala Spark for dataframes. (We've moved away from RDDs) However, if you use a lot of UDFs where Spark has to serialize your Python functions, you might consider rewriting those UDFs in a JVM language. Serialization overhead is still fairly substantial. Arrow is t…

Hi, Thanks for the answer. What you said resonates with me - with a few changes. Spark 2.3 will come with Arrow UDF, that should be a significant performance boost. In that way, yes - we are taking at a forward looking bet. About mllib - yes, we concur with you on algorithmic coverage. And yes, training is the major issue. For example, what I read of Uber's Michaelangelo infrastructure - it seems they train using Spa…

1) I've heard about vectorized Python UDFs in Spark 2.3. Thanks for reminding of that.

https://databricks.com/blog/2017/10/30/introducing-vectorize...

2) I'm not that familiar with what Uber is doing. My take is I'd like to use Spark for as much as I can, but there are parts that are either more performant or easier to accomplish in Python.

Spark with Arrow will definitely change the game.

Re: Ask HN: As a data scientist, what should be in my toolkit in 2018?

#147

Post by wishart_washy »

hobolord wrote: post #140
xitrium wrote: post #79

If you care about quantifying uncertainty, knowing about Bayesian methods is a good idea I don't see represented here yet. I care so much about uncertainty quantification and propagation that I work on the Stan project[0] which has an extremely complete manual (600+ pages) and many case studies illustrating different problems. Full Bayesian inference such as that provided by Stan's Hamiltonian Monte Carlo inference a…

do you have a recommended guide/textbook on learning stan? I've recently started doing more bayesian analysis, mainly bayesian estimation supercedes the t-test.

As someone who uses Stan - I would recommend reading the Stan reference documentation, it's essentially a textbook.

Also, get used to reading the Stan forums on Discourse. Happy Stanning

Re: Ask HN: As a data scientist, what should be in my toolkit in 2018?

#148

Post by nonbel »

vobios wrote: post #137
nonbel wrote: post #80

Earlier quoted context omitted.

Sometimes I think I'm the only one who isn't really a fan of the tidyverse. I've found it slower, more prone to dependency issues, more prone to silent errors, and less well documented than most R packages (ie most of what you find on CRAN).

> less well documented than most R packages I on the other hand, find most R packages provide barely readable documentation. I can just hope that the vignette exists and actually explains the inputs/outputs.

Here it is for one of the most often used functions:

https://www.rdocumentation.org/packages/ggplot2/versions/2.2...

You think this is better than barely readable?

Re: Ask HN: As a data scientist, what should be in my toolkit in 2018?

#149

Post by dxbydt »

> What’s the fizzbuzz test for data scientists anyway?

Here's 3 questions I was recently asked on a bunch of DS interviews in the Valley.

1. Probability of seeing a whale in the first hour is 80%. What's the probability you'll see one by the next hour ? Next two hours ?

2. In closely contested election with 2 parties, what's the chance only one person will swing the vote, if there are n=5 voters ? n = 10 ? n = 100 ?

3. Difference between Adam and SGD.

Re: Ask HN: As a data scientist, what should be in my toolkit in 2018?

#150

Post by ms013 »

maayank wrote: post #145
ms013 wrote: post #22

Mathematics. Which branch of math is domain dependent. Stats come up everywhere. Graphs do too. In addition to baseline math, you really need to understand the problem domain and goals of the analysis. Languages and libraries are just tools: knowing APIs doesn’t tell you at all how to solve a problem. They just give you things to throw at a problem. You need to know a few tools, but to be honest, they’re easy and you…

I have good background in graph theory (IMHO) but don't know many data science use-cases (I'm amateur at that). Could you point to some good start points?

Graphs show up all over the place. Social media: who is connected, which people interact. Cybersecurity: which computers/programs/users interact with which other computers/programs/users. Retail analytics: which products are bought with which other products; which products are more important in a graph than others.

Basically, any problem where you can establish relations between elements can be treated as a graph. I've used graphs for image analysis before too: pixels are vertices, edges represent neighborhood relations - especially useful when you make nonlocal connections (e.g., nonlocal means; graph-cut methods for segmentation; etc...)

I've worked with them in three of the above contexts: cybersecurity (my current projects), retail analytics, and image analysis. I've avoided social network stuff - never cared for that area much.

Post reply on HN