Full Stack Data Scientist (data janitor + data engineer + ML engineer + ML Ops + Business Analyst) is the future
Ugh, all this gets you is being mediocre at all of this.
We don't need data scientists, we need data engineers
41–50 of 367 posts
Re: We don't need data scientists, we need data engineers
#42Full Stack Data Scientist (data janitor + data engineer + ML engineer + ML Ops + Business Analyst) is the future
Maybe my sense of terminology is warped, but I always thought of
DataEngineer = DataJanitor
∪ MlEngineer
∪ MlOps
∪ BusinessAnalyst
Re: We don't need data scientists, we need data engineers
#43Good data is hard. Anyone conducting research in the physical sciences knows this firsthand. It takes painstaking effort to conduct carefully controlled experiments and collect a batch of good data that could then be used for analysis.
The promise of ML has always been to churn out good results from not-so-good data. If I now need to sanitize my data carefully, what's the advantage?
Re: We don't need data scientists, we need data engineers
#44What's the career trajectory for data engineers?
I enjoy the pipeline building and business stakeholder interfacing, but I'm not sure I want to be a SWE a decade from now...
Re: We don't need data scientists, we need data engineers
#45Earlier quoted context omitted.
It's not a straw man, I've seen it clear as day in several companies. When it comes to data science, it's "garbage in, garbage out". I've seen companies do lots of "data science" with a bunch of data scientists skilled in python and jupyter notebooks, only to discover a ton of work was useless because the incoming event data was tagged incorrectly due to a bug. The actual process of collecting, aggregating, cleaning…
>The actual process of collecting, aggregating, cleaning and verifying data is a hugely important skill, and not one I've really seen typical data scientists possess. Then they are not scientists. They have a label "scientist" but lack of rigor of actual science. I don't see why changing the label to "engineer" would suddenly make them have rigor.
[deleted]
Re: We don't need data scientists, we need data engineers
#46Full Stack Data Scientist (data janitor + data engineer + ML engineer + ML Ops + Business Analyst) is the future
These are incredibly disparate skill sets. Of course anyone would want to hire someone like this, and far more would claim to possess such a broad skill set, but in practice it is extremely rare. You'd need someone with excellent communication skills (presentation, memo writing, teamwork), project management skills (identifying & overcoming workflow bottlenecks), professional skills (timely responses, political savvy…
> You'd need someone with excellent communication skills (presentation, memo writing, teamwork), project management skills (identifying & overcoming workflow bottlenecks), professional skills (timely responses, political savvy), technical skills (application programming, advanced databases, advanced machine learning, Excel modeling) and finally some business domain knowledge.
This is pretty much the bare minimum requirement for any data scientist job I've ever interviewed for or held.
Re: We don't need data scientists, we need data engineers
#47Full Stack Data Scientist (data janitor + data engineer + ML engineer + ML Ops + Business Analyst) is the future
These are incredibly disparate skill sets. Of course anyone would want to hire someone like this, and far more would claim to possess such a broad skill set, but in practice it is extremely rare. You'd need someone with excellent communication skills (presentation, memo writing, teamwork), project management skills (identifying & overcoming workflow bottlenecks), professional skills (timely responses, political savvy…
exactly, these are characteristics of a unicorn and I think most of these skills are trivial to build up over time through practice and self-learning and these skills can yield great benefits both for employers and employees
Re: We don't need data scientists, we need data engineers
#48I always felt that tech-focused data scientists should also be required to know how process data end-to-end; at minimum, from a SQL database to deployed model, but knowing how to collect & clean data is important too. It seems like the industry is trying fill the gap that was created by a glut of people without math/cs backgrounds going into 5-week data science courses who then need hand-holding when they get real jo…
> Data science & engineering should be treated as a single collection of skill-sets.
This can be tough because there could be a lot in that skill set. You can't realistically expect someone to have solid knowledge of statistics including specialising in the sub-field and type of algorithms that your product needs, and also be able to write good code and act as a developer, and also have solid knowledge of all the tools for data streaming/processing/ETL. There is a point at which you're just stretching yourself too thin if you try to do all of these at once.
Of course, stuff like knowing how to interact with a database or employing good software development practices should be a very basic prerequisite and some scientists certainly shift things too far in the other direction and use their academic knowledge as an excuse to write poor code and not learn new tools.
I guess what I'm trying to say is that they are distinct skills but you still need all of them to some extent and striking the correct balance in one's skillset is really difficult.
Re: We don't need data scientists, we need data engineers
#49Preach! The data lifecycle is waaay overpopulated with Data Scientists who are not empowered or knowledgeable enough to work with product designers and engineers to do everything that empowers Data Science and ML. We need more Data Engineers involved at time zero in projects to help: 1. Plan out what data should be produced/captured by the product 2. Instrument systems to actually generate data consistently and effec…
This same sentiment (which I personally agree with) applies to software engineering. As in: engineers deliver more practical value than comp scientists. Now you can down-vote me to oblivion.
Re: We don't need data scientists, we need data engineers
#50Preach! The data lifecycle is waaay overpopulated with Data Scientists who are not empowered or knowledgeable enough to work with product designers and engineers to do everything that empowers Data Science and ML. We need more Data Engineers involved at time zero in projects to help: 1. Plan out what data should be produced/captured by the product 2. Instrument systems to actually generate data consistently and effec…
Who will do the proper cleaning then?
It doesn't matter, as long as you don't make the person with the PhD in biostatistics spend their time writing ETL pipelines, which is a wildly inefficient use of a very expensive resource.