Ask HN: As a data scientist, what should be in my toolkit in 2018? • Hacker News Board

Re: Ask HN: As a data scientist, what should be in my toolkit in 2018?

#171

Post by shepardrtc » Sun, Feb 25, 2018, 1:44 AM UTC

Xcelerate wrote: post #142

shepardrtc wrote: post #91

Earlier quoted context omitted.

Its designed to quickly weed out people who don't know the underlying math, just as FizzBuzz is designed to quickly weed out people who don't know programming.

I work as a data scientist, and my graduate research involved harmonic analysis over compact groups, optimization over Riemannian manifolds, and loopy belief propagation. You'd reject me in an interview because I couldn't remember the formula for standard deviation off the top of my head?

Personally, I don't ask weed-out questions. Never have, never will. I'm just saying that's what they're doing.

Re: Ask HN: As a data scientist, what should be in my toolkit in 2018?

#172

Post by red_hare » Sun, Feb 25, 2018, 7:16 PM UTC

dsacco wrote: post #109

red_hare wrote: post #69

Earlier quoted context omitted.

I'm a data engineer for a startup that's trying to hire its first data scientist. The range of candidates that apply with this title is massive. Defining our expectations has been challenging. My phone screen "fizzbuzz" is having them calculate a standard deviation from an array of data w/out with only basic operators (no numpy.std). Then explain why they choose population/sample and explain the difference. I studied…

> I studied math in undergrad so one of my requirements is "knows more math than me". What kind of questions are you asking to ensure that they’re correct when they’re speaking about math you don’t know?

This is a pretty hard problem I haven’t solved just yet. Generally, my in person interview is based on a set of DS problems I’ve been working on and had to do research myself to solve. What I look for is a strong intuition of the underlying math. Like, I can give them a formula and they can intuitively express what that means and explain it to me and then explain the next place they would take the solution. It’s not a perfect measure, but i’ve found the comfort with core concepts to be the most common trend among great data scientists I’ve worked with in the past.

Re: Ask HN: As a data scientist, what should be in my toolkit in 2018?

#173

Post by red_hare » Sun, Feb 25, 2018, 11:09 PM UTC

peatmoss wrote: post #114

red_hare wrote: post #69

Earlier quoted context omitted.

I'm a data engineer for a startup that's trying to hire its first data scientist. The range of candidates that apply with this title is massive. Defining our expectations has been challenging. My phone screen "fizzbuzz" is having them calculate a standard deviation from an array of data w/out with only basic operators (no numpy.std). Then explain why they choose population/sample and explain the difference. I studied…

I think regurgitation of math formulas is a terrible way to hire for most data science positions. I've seen a breakdown of data scientists into two categories: 1) People who are great at the mathematics behind the statistical tooling 2) People who are great at conceptualizing a relevant question, operationalizing it, and then using a computer to apply appropriate models. I think in most cases, for businesses needing…

I actually agree with your breakdown. But, as a "data engineer" with a math background who's spent 5 years building analytics tools, I already identify as your type-2. We're working in a field that already has a rich history of established statistics that needs to be interpreted and broken down, so I think we're looking for someone who's a type-1.

I do, however, think anyone with some lick of statistics background should know the formula for a standard deviation. Considering how fundamental the idea of variance is in statistics.

Re: Ask HN: As a data scientist, what should be in my toolkit in 2018?

#174

Post by red_hare » Sun, Feb 25, 2018, 11:12 PM UTC

Xcelerate wrote: post #142

shepardrtc wrote: post #91

Earlier quoted context omitted.

Its designed to quickly weed out people who don't know the underlying math, just as FizzBuzz is designed to quickly weed out people who don't know programming.

I work as a data scientist, and my graduate research involved harmonic analysis over compact groups, optimization over Riemannian manifolds, and loopy belief propagation. You'd reject me in an interview because I couldn't remember the formula for standard deviation off the top of my head?

It's not a hard weed-out for us. But, if you talk through variance for 10 minutes, you get pretty close to the formula for standard deviation.

Also, for our role, we're specifically hiring someone with extensive stats background since a large part of the role is learning domain-specific statistics of the industry we're targeting and figuring out how we can adopt those models with our data.

Re: Ask HN: As a data scientist, what should be in my toolkit in 2018?

#175

Post by red_hare » Sun, Feb 25, 2018, 11:21 PM UTC

arca_vorago wrote: post #82

red_hare wrote: post #69

Earlier quoted context omitted.

I'm a data engineer for a startup that's trying to hire its first data scientist. The range of candidates that apply with this title is massive. Defining our expectations has been challenging. My phone screen "fizzbuzz" is having them calculate a standard deviation from an array of data w/out with only basic operators (no numpy.std). Then explain why they choose population/sample and explain the difference. I studied…

I have to admit this scares me just a little bit. I'm a senior sysadmin who is trying to lateral transition into data science, but I'm no math whiz, I'm just good at pragmatic use of tech stacks and have a generally analytical mind. If you are a math undergrad how could I ever expect to know more math than you? Of course a standard deviation should be easy, but your comment on math just stuck out to me.

Honestly, the market is so oversaturated with PHDs who are switching to DS I don't see how anyone can transition into it from a different role. I'm speaking for myself as well as someone with a math background, most people don't consider a math degree and 4 years of applying math models as a data engineering experience enough to be a "data scientist". They just weed out anyone without a PHD.

But, what you're describing I would consider "data engineering" (at least how I have been hired to do it). Working through the business problems and pragmatically facilitating data, pipelines, databases, and models to solve those problems. It's less established and less "hot" but, IMO, it's a much more valuable job to most businesses.

Re: Ask HN: As a data scientist, what should be in my toolkit in 2018?

#176

Post by samuell » Mon, Feb 26, 2018, 1:41 PM UTC

ZeroCool2u wrote: post #43

chewxy wrote: post #42

My toolkit hasn't changed since 2016: - Jupyter + Pandas for exploratory work, quickly define a model - Go (Gonum/Gorgonia) for production quality work. (here's a cheatsheet: https://www.cheatography.com/chewxy/cheat-sheets/data-scienc... . Additional write-up on why Go: https://blog.chewxy.com/2017/11/02/go-for-data-science/ ) I echo ms013's comment very much. Everything is just tools. More important to understand t…

I'm a big Go fan, but this is the first time I've seen someone recommend Go for data science. After looking at this cheat sheet you've got me convinced though. Would you mind pointing me to any other less cheat sheet style and more in depth examples that you particularly like?