Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Some Reflections on Being Turned Down for a Lot of Data Science Jobs

tdhopper.com

Post reply on HN
Print view View original

11–20 of 175 posts

Re: Some Reflections on Being Turned Down for a Lot of Data Science Jobs

#11

Post by drxzcl »

throw_away_777 wrote: post #6
ska wrote: post #5

Earlier quoted context omitted.

In this particular case (data science) it is more an oversupply of candidates (qualified and not), plus difficulties defining and measuring "qualified", plus buzz. It's a difficult enough field to hire in when you understand what it is (and isn't) - and lots of companies are trying to do it with far more vague goals.

Why do you think candidates for data science jobs aren't qualified relative to other positions?

Not OP, but I think many companies aren't qualified to judge who is and who isn't a qualified candidate, at least for the first hires. This turns the whole thing into a "market for lemons".

I've helped a few organisations solve this bootstrap problem by helping out with candidate selection and interviews, but many other just don't ask for help.

Re: Some Reflections on Being Turned Down for a Lot of Data Science Jobs

#12

Post by halflings »

yummyfajitas wrote: post #9
throw_away_777 wrote: post #6

Earlier quoted context omitted.

Why do you think candidates for data science jobs aren't qualified relative to other positions?

I've probably interviewed about 70-100 such people in the past year and a half. Exactly 1 such person was qualified (I hired him). The issue in my view is the following: people who know both statistics and computer science are extremely rare. People who actually understand statistics are rare. I can probably weed out 1/3 to 1/2 of candidates simply by asking what a p-value is, or what precision/recall are (this inclu…

Ignoring what a p-value is does not mean that you don't know statistics. p-tests are not some inherent statistical property, they're just a useful model for significance. People coming from a CS background most likely didn't have to deal with p-values, but they can still be good at linear algebra or bayesian statistics.

(not sure I can defend somebody that does not know what precision/recall are)

Re: Some Reflections on Being Turned Down for a Lot of Data Science Jobs

#13

Post by drxzcl »

yummyfajitas wrote: post #9
throw_away_777 wrote: post #6

Earlier quoted context omitted.

Why do you think candidates for data science jobs aren't qualified relative to other positions?

I've probably interviewed about 70-100 such people in the past year and a half. Exactly 1 such person was qualified (I hired him). The issue in my view is the following: people who know both statistics and computer science are extremely rare. People who actually understand statistics are rare. I can probably weed out 1/3 to 1/2 of candidates simply by asking what a p-value is, or what precision/recall are (this inclu…

This is also very true! Usually we hire AI/stats folks and do a heck of a lot of training to get them up to speed on the development side of things. You can do it the other way around, but math is a lot harder to pick up outside of formal education than computer stuff.

Re: Some Reflections on Being Turned Down for a Lot of Data Science Jobs

#14

Post by motivic »

motivic wrote: post #10

This is a bit off-topic but one thing I'm curious is how the author manages to interview with these companies while holding down a full-time job and keep up-to-date with the latest developments. Interview for a data science position is typically a drawn-out process, with multiple rounds of interviews and possibly take-home projects. I found them to be very time and energy consuming. To share my story, I also had a di…

A better but more difficult approach is to distinguish oneself and have the companies go after you.

Re: Some Reflections on Being Turned Down for a Lot of Data Science Jobs

#15

Post by eanzenberg »

My analytical thought process on DS interviews:

- Signal is still quite low among noise, even with long multiple interviews, take-home homework, coding challenges, etc. Most relevant data is still hidden and takes months-years to come out.

- Companies seek to minimize false-positives much more than minimizing true-negatives.

- It's a numbers game from both ends because the probabilities are low, due to above 2 points.

Re: Some Reflections on Being Turned Down for a Lot of Data Science Jobs

#16

Post by ska »

throw_away_777 wrote: post #6
ska wrote: post #5

Earlier quoted context omitted.

In this particular case (data science) it is more an oversupply of candidates (qualified and not), plus difficulties defining and measuring "qualified", plus buzz. It's a difficult enough field to hire in when you understand what it is (and isn't) - and lots of companies are trying to do it with far more vague goals.

Why do you think candidates for data science jobs aren't qualified relative to other positions?

Two main factors make data science stick out a little for me right now, although it isn't unique.

One is that there is buzz & excitement around "data science" right now. Nothing specific to this area, but in my experiences this creates a large number of under- or un-qualified applicants. It also creates an environment for companies to desire to hire a role they are not well qualified to hire for. It is really difficult to hire well for roles you don't understand well.

The second thing is that extremely few people are actually ready for this sort of job straight out of an academic program. A related Ph.D. or post doc plus a few years solid training in industry can make you a great candidate, but the academic work alone usually isn't even remotely close. There is confusion about this among both candidates (don't know what they don't know) and hiring managers (don't know what they are actually looking for).

Add to that an oversupply of academic credentials relative to academic jobs and you have a problem. If you are a large company with a well defined data science program and a defined "entry level" data science role, if you take skill development and training seriously and have the senior staff for it, well then you are fine taking strong academic candidates and turning them into talented data scientists. If you are a less experienced company looking for scientists to solve a problem you don't fully understand, you may be in for a pretty rough ride.

Re: Some Reflections on Being Turned Down for a Lot of Data Science Jobs

#17

Post by achompas »

ska wrote: post #5
pascalxus wrote: post #3

The real reason you were rejected, or anyone applying in a highly competitive field: an oversupply of qualified candidates.

In this particular case (data science) it is more an oversupply of candidates (qualified and not), plus difficulties defining and measuring "qualified", plus buzz. It's a difficult enough field to hire in when you understand what it is (and isn't) - and lots of companies are trying to do it with far more vague goals.

I think this is closer to reality. Tim is not competing with many folks like him -- he's a knowledgable, experienced, and capable data scientist with significant infrastructure experience, and a net positive to any team he joins. Some problems possibly originating from the company perspective

(1) They are inundated with applications folks of all sorts of backgrounds: engineering, finance, academia, marketing, BI/analytics, etc.

(2) They still haven't figured out hiring. To be fair, no one really has figured it out. Jeff Kolesky recently covered this as part of an excellent blog post. [0]

(3) In addition to the typical variance in engineering interview processes, we now introduce variance in the definition of data science across companies, which just complicates things further.

(4) Basically everything else Tim mentioned in his post: role or goals aren't clearly defined, remote data science is an unknown, etc.

[0] http://kolesky.com/datums/job-search/

Re: Some Reflections on Being Turned Down for a Lot of Data Science Jobs

#18

Post by SatvikBeri »

yummyfajitas wrote: post #9
throw_away_777 wrote: post #6

Earlier quoted context omitted.

Why do you think candidates for data science jobs aren't qualified relative to other positions?

I've probably interviewed about 70-100 such people in the past year and a half. Exactly 1 such person was qualified (I hired him). The issue in my view is the following: people who know both statistics and computer science are extremely rare. People who actually understand statistics are rare. I can probably weed out 1/3 to 1/2 of candidates simply by asking what a p-value is, or what precision/recall are (this inclu…

I have similar experiences. To add some color: I find that for data science tasks, someone who knows statistics & can program is much, much more productive than someone who only knows one. Part of that is because data science has to do their own product management–the question you ask next changes quite rapidly depending on the results of a single query.

That said, most companies should probably be hiring data engineers rather than data scientists–for most "data science" jobs I've seen, almost no statistics is actually necessary/useful.

Re: Some Reflections on Being Turned Down for a Lot of Data Science Jobs

#19

Post by x0x0 »

ryandrake wrote: post #4
pascalxus wrote: post #3

The real reason you were rejected, or anyone applying in a highly competitive field: an oversupply of qualified candidates.

But, wait! I thought there was a "shortage of tech workers."

Which must necessarily be urgently addressed by open immigration for anyone who can write code! The future of silicon valley demands it!

Re: Some Reflections on Being Turned Down for a Lot of Data Science Jobs

#20

Post by dwpdwpdwpdwpdwp »

yummyfajitas wrote: post #9
throw_away_777 wrote: post #6

Earlier quoted context omitted.

Why do you think candidates for data science jobs aren't qualified relative to other positions?

I've probably interviewed about 70-100 such people in the past year and a half. Exactly 1 such person was qualified (I hired him). The issue in my view is the following: people who know both statistics and computer science are extremely rare. People who actually understand statistics are rare. I can probably weed out 1/3 to 1/2 of candidates simply by asking what a p-value is, or what precision/recall are (this inclu…

I'm surprised people bother spending so much energy looking for someone who is both a statistician and a computer scientist knowing they are so rare. There are so many more statisticians who can at least communicate and work effectively with developers and vice versa. Why not just compose a team? I feel like just like other professionals have assistants, statisticians should have them too, and they'd be focused on the computer science and deployment of the applied statistics.
Post reply on HN