Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Beware the data science pin factory: The power of the data science generalist

multithreaded.stitchfix.com

Post reply on HN
Print view View original

21–30 of 79 posts

Re: Beware the data science pin factory: The power of the data science generalist

#21

Post by jonathankoren »

perpetualpatzer wrote: post #16
gowld wrote: post #8

Earlier quoted context omitted.

Infrastructure isn't your product. Why build an infrastructure instead of buying it?

Not OP, but given the context, it seems OP is using infrastructure to mean "all prerequisites to doing ML/data analysis work." Some of that (e.g. datawarehousing, etc.) is easier to outsource; other parts (data acquisition from your product, ETL design, etc.) are necessarily bespoke to your company an thus not readily "buyable." I understand OP to be arguing roughly "you can get a good DBA for much cheaper than you c…

You have correctly understood what I am saying.

Re: Beware the data science pin factory: The power of the data science generalist

#22

Post by steve76 »

reilly3000 wrote: post #7

I really wish hiring managers read this. I am a data generalist, and have had no traction with obtaining even an interview for a data science job. I’ve setup a private JupyterHub where I run python ETL, interactive models, and dashboards. I deployed Metabase several times and have written hundreds of SQL queries. I’ve used Tableau with gigantic datasets. I built a front end serverless analytics pipeline from scratch…

These are day jobs no one really wants to do. Punch a clock, take the check to pay your bills, and get out to do your own thing.

Find out what they are looking for already before the interview, and just walk in and open with something like:

"So ... ... NimbusML, right???

... ...

Good!"

If you can't even do that, hang out around people at software meetups. Spend maybe a month or two not saying anything and let them wonder about you. Be nice, give work away for free, buy a few drinks or bring food, and then say you have to quit programming because you have to work to pay your bills. If they let you fall flat, do it to another group of people. Eventually the odds will work out.

When you get the job, just don't be a problem. Get things done the easiest way possible without being a headache. Think of a co-pilot allowing a pilot to checkoff a list of procedures. That's what they want.

Re: Beware the data science pin factory: The power of the data science generalist

#23

Post by opportune »

reilly3000 wrote: post #7

I really wish hiring managers read this. I am a data generalist, and have had no traction with obtaining even an interview for a data science job. I’ve setup a private JupyterHub where I run python ETL, interactive models, and dashboards. I deployed Metabase several times and have written hundreds of SQL queries. I’ve used Tableau with gigantic datasets. I built a front end serverless analytics pipeline from scratch…

The one thing I see kind of missing is a math background or at least a project proving that that is in your skillset (recommendations sounds like it could fit this). There are a lot of people with a similar background to you and normally those are in "business intelligence/analytics" or "data engineering" where they are mostly writing sql queries and interacting with dashboards/OLAP cubes or setting up those dashboards/cubes.

That's perfectly fine but it's not what traditionally is referred to as data science. I'm actually quite annoyed at what has been happening to the term data science lately - it's supposed to be some stats-heavy/applied-AI role but a lot of companies hiring "data scientists" are really just hiring SQL jockeys.

Personally I've done both data science and data infrastructure and I like infrastructure a lot more anyway. And it sounds like you are somewhat qualified for that with some of your pipeline work (although big data experience is also important). A LOT of data science departments have no idea what type of business value they are supposed to be adding, are doing shitty boring work with glorified titles, or are improperly integrated with the company at large (bad productionizing processes, poor data infrastructure). There's always going to be a need for data infrastructure but the "data science" hype is going to fade once all the shitty data departments cut the fat.

Re: Beware the data science pin factory: The power of the data science generalist

#24

Post by lifeisstillgood »

apohn wrote: post #19
reilly3000 wrote: post #7

I really wish hiring managers read this. I am a data generalist, and have had no traction with obtaining even an interview for a data science job. I’ve setup a private JupyterHub where I run python ETL, interactive models, and dashboards. I deployed Metabase several times and have written hundreds of SQL queries. I’ve used Tableau with gigantic datasets. I built a front end serverless analytics pipeline from scratch…

Have you tried paid services/consulting arms of software or cloud companies? Teams that bill customers at an hourly rate? They generally look for generalists who can help customers tackle problems at different levels of the stack. They aren't looking for PhDs in statistics. When I interview people who have your type of background, I tend to get confused by what exactly it is the person wants to do (Analyze Data? Buil…

I don't see this as a bad thing - it's a lifecycle thing. I absolutely would want someone like the GP to start my data team from day one - there is a lot to build and much to hang together.

When I am up and running I don't want yet another generalist - or rather I will happily take one, I just will put them in a box making pins.

Perhaps the GP will do better at the consulting level - or even some level of productise consulting - and out of the box product

Re: Beware the data science pin factory: The power of the data science generalist

#25

Post by kradroy »

reilly3000 wrote: post #7

I really wish hiring managers read this. I am a data generalist, and have had no traction with obtaining even an interview for a data science job. I’ve setup a private JupyterHub where I run python ETL, interactive models, and dashboards. I deployed Metabase several times and have written hundreds of SQL queries. I’ve used Tableau with gigantic datasets. I built a front end serverless analytics pipeline from scratch…

I'm a hiring manager who shares this view. I see three distinct but overlapping skillsets with creating machine learning - as they mention in the article: data engineering, data science, ML engineering. I would never hire a person who only had one of those skillsets. I prefer all three, but can settle for two depending on the situation.

The trouble you might be having in getting an interview is probably partly to do with your background and likely also that those job postings get A LOT of submissions. Other hiring managers in my department as well as myself have found we get 10x more submissions for DS/ML positions than software dev positions. In general it's a really unrefined and new job skills that anyone and everyone who's taken a coursera course in regression or clustering will apply.

Re: Beware the data science pin factory: The power of the data science generalist

#26

Post by opportune »

I agree and disagree with this post. I do think data scientists need to be better at data processing and do more of it. But I still think you do need a separation of labor between people setting up pipelines and people building models from the data. The real issue is that there are a lot of data science departments where they wittle away at their models in some notebook and then they're "done" once the notebook is showing the right metrics. Data scientists should be writing their models from the beginning so that they can productionize them once they are finished. There shouldn't be frequent hand off events requiring lots of communication between DS, pipelines, and data engineering teams, there should be an integration process set up so the flow of work continues to function without intervention.

Re: Beware the data science pin factory: The power of the data science generalist

#27

Post by rchaud »

apohn wrote: post #19
reilly3000 wrote: post #7

I really wish hiring managers read this. I am a data generalist, and have had no traction with obtaining even an interview for a data science job. I’ve setup a private JupyterHub where I run python ETL, interactive models, and dashboards. I deployed Metabase several times and have written hundreds of SQL queries. I’ve used Tableau with gigantic datasets. I built a front end serverless analytics pipeline from scratch…

Have you tried paid services/consulting arms of software or cloud companies? Teams that bill customers at an hourly rate? They generally look for generalists who can help customers tackle problems at different levels of the stack. They aren't looking for PhDs in statistics. When I interview people who have your type of background, I tend to get confused by what exactly it is the person wants to do (Analyze Data? Buil…

> I tend to get confused by what exactly it is the person wants to do (Analyze Data? Build an Analytics Pipeline/Architecture? Write Software/Services?

But this is the crux of the job-seeker's dilemma. If he/she is specific about their interests when speaking to an interviewer, they might get a response like "well, we're really looking for someone whose operational focus is [something else]".

And if they're not super-specific (I doubt anyone does data analysis exclusively without any other involvement in the project), but instead attempt to give examples where they had demonstrable impact working across a number of domains, you might hear a response like this:

> Even reading your comment, you don't sound like somebody who wants to analyze data.

Re: Beware the data science pin factory: The power of the data science generalist

#28

Post by sgt101 »

zwieback wrote: post #13
sgt101 wrote: post #3

A very good article, but I think that there is a missing concept - which is organisational maturity. In a fully mature data driven organisation (like... errm Google I guess - reading Jeff Deans papers anyway) there is a well developed data fabric, polished processes for providing credentials and authority, right sized resourcing pools and also substantial diversity of specialisation coupled with experience and domain…

> A very good article, but I think that there is a missing concept - which is organisational maturity. Maturity and also scale - I suppose a small or even one-man shop requiring a generalist could be mature. Once you get to a certain size specialization happens automatically.

I agree that a one man shop can be "mature"; but there are many very large scale operations that have cultures that absolutely preclude speciality.

Re: Beware the data science pin factory: The power of the data science generalist

#29

Post by sidlls »

reilly3000 wrote: post #7

I really wish hiring managers read this. I am a data generalist, and have had no traction with obtaining even an interview for a data science job. I’ve setup a private JupyterHub where I run python ETL, interactive models, and dashboards. I deployed Metabase several times and have written hundreds of SQL queries. I’ve used Tableau with gigantic datasets. I built a front end serverless analytics pipeline from scratch…

None of what you describe is something I'd hire into a data science role specifically. Some of those skills are skills I'd expect a data scientist to have (e.g. SQL skills). A data scientist in this context has to have an understanding of basic statistics generally: hypothesis testing, modelling techniques and their applications, and performance tuning and evaluation. They also need to understand how to devise and run experiments that collect and make use of data in practice (i.e. "real world" data). I wouldn't necessarily expect a candidate to be able to derive the formulas involved, but it would be the odd candidate who truly grasped the nuances who could not do so.

Re: Beware the data science pin factory: The power of the data science generalist

#30

Post by mrmrcoleman »

reilly3000 wrote: post #7

I really wish hiring managers read this. I am a data generalist, and have had no traction with obtaining even an interview for a data science job. I’ve setup a private JupyterHub where I run python ETL, interactive models, and dashboards. I deployed Metabase several times and have written hundreds of SQL queries. I’ve used Tableau with gigantic datasets. I built a front end serverless analytics pipeline from scratch…

We’re hiring.

Email me: mark at dotscience dot com

Post reply on HN