Data Science: Reality Doesn't Meet Expectations
1–10 of 168 posts
Re: Data Science: Reality Doesn't Meet Expectations
#2From my perspective as a data person, everything on this list is true. I would add on to #4 to say "You're likely the only data person" to say "You're likely the only data person and expected to do everything you need to do your job yourself" (from sourcing the data to deploying your model).
High Data Scientist salaries and expectations combined with a shortage of qualified people often mean you're expected to be a one-person band, which I find to be miserable.
Re: Data Science: Reality Doesn't Meet Expectations
#3This rings very true to me. I'm working on moving over to an SWE role in the next few years for many of these reasons.
I'll just add one: the business absolutely doesn't care how you get your answer, only if they're reliable enough (hand grenade close is better than most companies have today).
While this seems obvious enough to anyone with a few years under their belt, to the new DS grad who has their time series analysis canned in favor of slapping a simple moving average in place and shipping it can be rather disillusioning.
Re: Data Science: Reality Doesn't Meet Expectations
#4Teams being small, data being crummy, infra being hard, and yet expectations being high aren't so much complaints as the they are the job description.
The point of data scientists and the related roles listed in the article are not to just churn out the fun stuff, but to wade through the institutional and technical muck and mire it takes to bring the fun stuff to bear on a relevant business problem and to communicate the results in a way that people of all walks can understand.
Re: Data Science: Reality Doesn't Meet Expectations
#5> Moreover, you may quickly realize much of this work is repetitive and while time-consuming, is “easy”. In fact, most analyses involve a great deal of time to understand the data, clean it and organize it. You may spend a minimal amount of time doing the “fun” parts that data scientists think of: complex statistics, machine learning and experimentation with tangible results.
This. Universities and online challenges provide clean labeled data, and score on model performance. The real world will provide you... “real data” and score you (hopefully) by impact. Real data work requires much more than modeling. Understanding the data, the business and value you create are important.
As per #6, better data and model infrastructure is crucial in keeping the time spent on these activities manageable, but I do think they’re important parts of the job.
I’ve seen data science teams at other companies working for years on topics that never see production because they only saw modeling as their responsibility. Even the best data and infrastructure in the world won’t help if data scientists do not feel co-responsible for the realization of measurable value for their business.
Training integrative data professionals could be a great opportunity for bootcamps. Universities will (understandably) focus on the academically interesting topic of models, while companies will increasingly realize they need people with skills across the data value chain. I know I would be interested in such profiles. :)
Re: Data Science: Reality Doesn't Meet Expectations
#6I've been a data person for the past year and a half and I'm very disappointed with the bewildering array of titles out there and the rather vague meanings behind them (Data Analyst, Data Scientist, Data Engineer, ML Engineer).
It's overall hurting my ability to build my personal brand and seek roles that are a fit for my existing skillset and aspirations.
What exactly does 'ML Engineer' communicate to employers in terms of baseline skills? Is the role closer to that of a data engineer or an analyst?
Re: Data Science: Reality Doesn't Meet Expectations
#7It works well when subject matter experts exist in the org and collaborate/supervise/drive data folk, to solve some issue the sme's have spent enough of their own time thinking about.
If its just data folk by themselves getting dumped with org data and told to find pirate gold...then its a crap shoot.
Re: Data Science: Reality Doesn't Meet Expectations
#8Teams being small, data being crummy, infra being hard, and yet expectations being high aren't so much complaints as the they are the job description. The point of data scientists and the related roles listed in the article are not to just churn out the fun stuff, but to wade through the institutional and technical muck and mire it takes to bring the fun stuff to bear on a relevant business problem and to communicate…
[deleted]
Re: Data Science: Reality Doesn't Meet Expectations
#9As a data dude in public/nonprofit healthcare-landia I agree with all this, plus:
- It's essential to have/develop domain expertise in your industry.
- Beware plausible, but incorrect (or poorly interpreted) data that supports yours (or others') assumptions/biases.
- Add on to #4 - at least as bad as this is having well-intentioned people on your team who "know enough (a bit of SQL or low/no-code data tool") to be dangerous. Um, why are you joining unnecessary tables, or using a different alias for the same columns/tables in different queries, with no comments or standard formatting?
- Hold your nose, but anything you do in SQL/R/Python/even fancier programming tool/language is going to pass through MS Excel at least once sooner or later which can irreversibly bastardize CSVs (even just opening without saving!), truncate precision to 15 digits, change data types, etc.
- So glad for the callout in #7 - there are clearly devs/data folks out there who are happy to take on an "interesting programming project at a great paying job" - that isn't serving the best interests of humanity.
Re: Data Science: Reality Doesn't Meet Expectations
#10I've been a data person for the past year and a half and I'm very disappointed with the bewildering array of titles out there and the rather vague meanings behind them (Data Analyst, Data Scientist, Data Engineer, ML Engineer). It's overall hurting my ability to build my personal brand and seek roles that are a fit for my existing skillset and aspirations. What exactly does 'ML Engineer' communicate to employers in t…
I've been working in data roles for 10 years and hold a masters in ML. I've hired and managed each of the roles you mentioned. I think of the responsibilities of each of those roles as:
-ML Engineers as building software infrastructure to scale machine learning inference and training.
-Data engineers focusing on data infrastructure and pipelining into either model inference, training, or other business intelligence platforms
-Analysts consume the product of the data engineer in the BI platform or excel, where the results would be consumed as a report in some form.
-And ML Researchers would be those inventing novel machine learning algorithms to deploy in the ML Infrastructure managed by the ML Engineers
-And data scientists to deploy well-known ML algorithms or statistical inference on varying datasets on the ML Infrascturue or as a slide deck.