Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Data Science: Reality Doesn't Meet Expectations

dfrieds.com

Post reply on HN
Print view View original

71–80 of 168 posts

Re: Data Science: Reality Doesn't Meet Expectations

#71

Post by proverbialbunny »

tqi wrote: post #39
kristjansson wrote: post #4

Teams being small, data being crummy, infra being hard, and yet expectations being high aren't so much complaints as the they are the job description. The point of data scientists and the related roles listed in the article are not to just churn out the fun stuff, but to wade through the institutional and technical muck and mire it takes to bring the fun stuff to bear on a relevant business problem and to communicate…

Yeah this guy seems to think Data Science work should be like doing a problem set for CS class. Sorry that you have to deal with messy data, fragile infra, and limited resources - I know it's not "fun", but frankly that's what the money is for.

The author wants to be an MLE but doesn't know it.

Re: Data Science: Reality Doesn't Meet Expectations

#72

Post by FridgeSeal »

> You’re likely the only “data person... Because people don’t know what data science does, you may have to support yourself with work in devops, software engineering, data engineering, etc.

Nothing has summed up my entire working experience more than this, it’s almost painfully accurate.

On one hand it’s an exciting challenge, you learn a lot and you get good at adapting to these situations.

On the downside I have practically no senior data science people to turn to for help when I do need it, which is frustrating.

Re: Data Science: Reality Doesn't Meet Expectations

#73

Post by ims »

proverbialbunny wrote: post #64
hectormalot wrote: post #5

> Moreover, you may quickly realize much of this work is repetitive and while time-consuming, is “easy”. In fact, most analyses involve a great deal of time to understand the data, clean it and organize it. You may spend a minimal amount of time doing the “fun” parts that data scientists think of: complex statistics, machine learning and experimentation with tangible results. This. Universities and online challenges…

>You may spend a minimal amount of time doing the “fun” parts that data scientists think of: complex statistics, machine learning and experimentation with tangible results. I don't get why building a model people consider to be the "fun" part. That's mostly spitting data in, watching a loading screen, and then observing the output. That's not fun, that's boring. The fun part is looking at the data and gleaming all th…

The standard whatever.fit(X, y) isn't very appealing but there are much more bespoke models that require creative engagement with stats/CS knowledge, e.g. Bayesian hierarchical models or deep learning models that are more complicated than what can be copy/pasted from Medium.

Re: Data Science: Reality Doesn't Meet Expectations

#74

Post by proverbialbunny »

eanzenberg wrote: post #63

I guess I'm in the minority in these threads..? I've been doing machine learning / model-building / pushing models to prod and maintaining for about 6 years now. It's still 50/50 understanding the data and building/tweaking/training/testing models. But it sounds like most people with this title are analysts? At least that's what posts and threads lead me to believe. I've also met a lot of people with titles like "ML…

Pushing models to prod is often MLE work depending on the organization, though MLE is often slang like how dev is slang. MLE an be a job title just as Developer can be a job title, but more common than not the job title is Machine Learning Software Engineer, or just Software Engineer for short.

I suspect a lot of people want the sexy Data Science job title, which is why there has been such a push for it, and why most new "data scientists" take the title but do Data Engineer / Infrastructure Software Engineer work or MLE work instead.

I think MLE is more sexy in a lot of ways, and it often pays better than DS work, so it's odd that many haven't flocked to that job title, but maybe the whole software engineer part turns people off for some sort of reason.

Me, I'm more a classic data scientist / research engineer, which involves a lot of digging through data and research and generalized learning, then presenting my findings. I'm not using any ML on the job right now, but often I have in the past. It's just a tool, not an end.

Re: Data Science: Reality Doesn't Meet Expectations

#75

Post by ajeet_dhaliwal »

Point 5, “ Your impact is tough to measure” is also shared by Quality Engineering and SRE, and not unique to Data Science. The point about being a support role holds true for them and it is thoroughly frustrating when a front-end dev makes a small change to a visual element is praised to the roof while complex automation projects by the quality team, ingenious recovery and reliability projects by SRE, and massive and fascinating inferences by data science are undervalued by leadership. The truth is most leaders just can’t connect the dots. I’ve worked as a full stack engineer btw do not taking a dig at front-end work, but it’s clearly easier to measure impact. I’ve worked in quality too and when you’re only called in to ask why one bug got out and never asked about the thousands you’ve stopped it’s demoralizing. It’s part of the reason I started Tesults (https://www.tesults.com), if you’re in one of these support roles, measure, measure and measure and throw those reports into the faces of leadership. It shouldn’t have to be done but without it, the point the author is making here will take place.

Re: Data Science: Reality Doesn't Meet Expectations

#76

Post by proverbialbunny »

stuxnet79 wrote: post #60
aeyes wrote: post #52

Earlier quoted context omitted.

Depends on the amount of data, reports, pipelines... If the company is small you might not have any of these problems. Every Mom&Pop store has some sort of data to run the business but they don't need a "data" person. Once you have 10s of datastores + pipelines, 100s of reports and a "data lake" in the TBs you'll likely be needing specialized people.

So far I've spent my career in small teams / startups and it's starting to become apparent that a lot of what's assumed in these titles only applies in larger corporations where resources are abundant and it makes business sense to have a specialist focused on a single aspect. Unfortunately I'm at a point where I have 'jack of all trades master of none' syndrome and it's causing me to fall in between the cracks profe…

ymmv, but as a data scientist at young startups, I often am the one giving new tasks to the software engineers, and facilitate teaching and training if they need help.

Most of those roles a software engineer can do.

Re: Data Science: Reality Doesn't Meet Expectations

#77

Post by Der_Einzige »

Icathian wrote: post #3

This rings very true to me. I'm working on moving over to an SWE role in the next few years for many of these reasons. I'll just add one: the business absolutely doesn't care how you get your answer, only if they're reliable enough (hand grenade close is better than most companies have today). While this seems obvious enough to anyone with a few years under their belt, to the new DS grad who has their time series ana…

Nothing reliably consistently beats ARIMA models in time series forecasting to this day

That's pretty sad when you think about it, but it's painfully true.

Re: Data Science: Reality Doesn't Meet Expectations

#78

Post by FridgeSeal »

proverbialbunny wrote: post #64
hectormalot wrote: post #5

> Moreover, you may quickly realize much of this work is repetitive and while time-consuming, is “easy”. In fact, most analyses involve a great deal of time to understand the data, clean it and organize it. You may spend a minimal amount of time doing the “fun” parts that data scientists think of: complex statistics, machine learning and experimentation with tangible results. This. Universities and online challenges…

>You may spend a minimal amount of time doing the “fun” parts that data scientists think of: complex statistics, machine learning and experimentation with tangible results. I don't get why building a model people consider to be the "fun" part. That's mostly spitting data in, watching a loading screen, and then observing the output. That's not fun, that's boring. The fun part is looking at the data and gleaming all th…

MLE is a fairly new title that, as best I can tell, exists primarily in those few places that have a mature enough workflow to have people who can actually dedicate their time to the ML part and have other roles take care of the rest.

Everywhere else, there is only DS, and it involves everything.

To answer your first question though, the training and testing of these models is fun because it feels like a puzzle game: did all my understanding and preparation of the data (and the business) pay off and the model does its job as expected? Is there something I’m missing? What’s the simplest model + configuration I can use that produces acceptable results and what does that say about the problem space? Can I combine models in some way to get the results? Is nothing working because it’s an ultimately fruitless exercise and our hypothesis is wrong? Or is there something we’re missing that is in turn the reason the model is missing something? Etc etc.

Then as the output you get something that ingests some data and then makes a decision with it! That’s cool to me.

Re: Data Science: Reality Doesn't Meet Expectations

#79

Post by apohn »

throwaway713 wrote: post #54

As a research-oriented data scientist at one of the larger tech companies, I can confirm that even here, a lot of people are unsure about what exactly data scientists are supposed to do. My most frequent request is "tell us why metric X dropped", to which the answer is often a subtle combination of many different factors (often random fluctuation) that doesn't lead to a pleasing actionable result in the sense of "her…

>But in terms of career progression and job safety, the risk is just way too high, at least for me personally. I save the highly mathematical stuff for a hobby.

I think the sad truth is that this is the reality of work no matter if you are a Data Scientist or not. What you thought you would be doing to show your worth and climb the ladder gets blurred in with KPIs you didn't set, politics you didn't create, goals and deadlines you had no input into, etc. One of the unique challenges you can face as a Data Scientist is that you may interface with people in many different groups, all of which have different goals which may be in conflict with each other. Compare this to other roles where you ultimately only follow the goals of the organization you report into.

Re: Data Science: Reality Doesn't Meet Expectations

#80

Post by FridgeSeal »

ajeet_dhaliwal wrote: post #75

Point 5, “ Your impact is tough to measure” is also shared by Quality Engineering and SRE, and not unique to Data Science. The point about being a support role holds true for them and it is thoroughly frustrating when a front-end dev makes a small change to a visual element is praised to the roof while complex automation projects by the quality team, ingenious recovery and reliability projects by SRE, and massive and…

> and it is thoroughly frustrating when a front-end dev makes a small change to a visual element is praised to the roof while complex automation projects by the quality team, ingenious recovery and reliability projects by SRE, and massive and fascinating inferences by data science are undervalued by leadership.

I feel this in my bones lol.

The frustration when the results of weeks/months of hard work are glossed over with a “oh that’s nice” in favour of endless praise for the front end team putting a picture backdrop on the search page or something.

Didn’t matter how many times we sold them on the benefits, or explained the work that went into it (at both executive summary level and detail) or did all those things you’re supposed to do, if it was more than one step away from directly causing it, or slightly more abstract than “we moved the button” it was wasted on leadership/management.

Spent a couple of weeks fixing data pipelines and ETL/database infrastructure and processes and now everything runs faster, and runs on a smaller and cheaper cluster and as a result managed to put together some analysis and modelling on customer behaviour that shows if you do xyz you’d expect to see uptick in this thing. Doesn’t matter, Bob changed where the button sits and we saw 20% more sales, good job Bob, everyone: be more like Bob.

Post reply on HN