I'm a SWE and data engineering actually sounds super interesting to me. Unfortunately, my day-to-day doesn't provide opportunities to work with the massive amounts of data we generate. I've looked into learning this stuff online but courses like DataCamp seem too basic (I have experience with Python and data cleaning in a research setting along with some academic ML experience) or downright a bit scammy. Many of the articles I read online about this also frame data engineering as a way to transition into the tech industry, which isn't my blocker. Does anyone have advice to help me transition away from pure software to a job as a data engineer?
We don't need data scientists, we need data engineers
61–70 of 367 posts
Re: We don't need data scientists, we need data engineers
#62I want to live in a world where data scientists making nearly $500K can understand and correctly implement simple concepts such as fixed effects. Is that asking for too much?
Sure, but what about at $80k?
None of this discussion is helped by the fact that companies want to get into data science without actually having much data strategy.
Re: We don't need data scientists, we need data engineers
#63Full Stack Data Scientist (data janitor + data engineer + ML engineer + ML Ops + Business Analyst) is the future
These are incredibly disparate skill sets. Of course anyone would want to hire someone like this, and far more would claim to possess such a broad skill set, but in practice it is extremely rare. You'd need someone with excellent communication skills (presentation, memo writing, teamwork), project management skills (identifying & overcoming workflow bottlenecks), professional skills (timely responses, political savvy…
I've seen this in leadership who want to move to "Devops", its the classic "if we find this one person who can do everything we will have no problems!"
The reality is of course, nobody can be amazing at the full lifecycle of an application. Some do better in infra, some better in backend, front end, etc.
A successful leader must find what is needed for the product/application pipeline and hire appropriate skill sets, trying to find the one candidate to rule them all is giving up on planning IMO.
Re: We don't need data scientists, we need data engineers
#64Full Stack Data Scientist (data janitor + data engineer + ML engineer + ML Ops + Business Analyst) is the future
These are incredibly disparate skill sets. Of course anyone would want to hire someone like this, and far more would claim to possess such a broad skill set, but in practice it is extremely rare. You'd need someone with excellent communication skills (presentation, memo writing, teamwork), project management skills (identifying & overcoming workflow bottlenecks), professional skills (timely responses, political savvy…
As the saying goes:
If you're looking for a data scientist with XYZABC skills, that's not a data scientist, that's a data science team.
Re: We don't need data scientists, we need data engineers
#65Earlier quoted context omitted.
These are incredibly disparate skill sets. Of course anyone would want to hire someone like this, and far more would claim to possess such a broad skill set, but in practice it is extremely rare. You'd need someone with excellent communication skills (presentation, memo writing, teamwork), project management skills (identifying & overcoming workflow bottlenecks), professional skills (timely responses, political savvy…
exactly, these are characteristics of a unicorn and I think most of these skills are trivial to build up over time through practice and self-learning and these skills can yield great benefits both for employers and employees
Or you can recognize that they're the characteristics of a unicorn and split the role into multiple positions.
Re: We don't need data scientists, we need data engineers
#66Having to deal with data scientists, I absolutely agree. The thing that I've seen that lands in the "lab" vs production distinction is that these people expect their data to be pristine. They flip out when the world isn't as perfect as their models want. Leads to me as just a normal software developer having to do the data analysis and figure out how to clean it up. I also end up having to be the one to talk to data…
I think that this is more of a problem with the specific people that you have worked with and it isn't inherent to the role of a data scientist.
It’s becoming more inherent, especially as the field is populated with people who have no experience with the “science” part. That is, with the very real and ubiquitous problem of collecting and cleaning data to make it fit for scientific study. Even theoretical physicists, for example, participate in and rely on empirical data collection, and understand deeply how messy and fraught with error it is.
I don’t see the same appreciation or consideration in general in the field of data “science.”
Re: We don't need data scientists, we need data engineers
#67I'm a SWE and data engineering actually sounds super interesting to me. Unfortunately, my day-to-day doesn't provide opportunities to work with the massive amounts of data we generate. I've looked into learning this stuff online but courses like DataCamp seem too basic (I have experience with Python and data cleaning in a research setting along with some academic ML experience) or downright a bit scammy. Many of the…
We just hired a SWE turned Data Engineer. You don't need to handle massive data to make the transition. I believe all the person did was build a small but robust pipeline that took some API response data, cleaned it, populated some sqlite dbs, replicated it for a small team to use and kept it updated every few days automatically.
Re: We don't need data scientists, we need data engineers
#68Having to deal with data scientists, I absolutely agree. The thing that I've seen that lands in the "lab" vs production distinction is that these people expect their data to be pristine. They flip out when the world isn't as perfect as their models want. Leads to me as just a normal software developer having to do the data analysis and figure out how to clean it up. I also end up having to be the one to talk to data…
Just want to say that while the data science profession definitely includes a wide range of people and skillsets, a good data scientist should be practical and able to work with the available data in whatever state it's in.
No good data scientist should ever expect data to be pristine. And a good data scientist, even if they don't have quite the engineering chops necessary to build a production-quality ETL, should know enough about the process to help guide it. If they aren't a part of that process, they're not being a good DS. They can't expect someone not involved with their problem to know what tradeoffs to make, and if they don't know exactly how their data went from raw form to the ETL-ed form, they're probably going to make bad assumptions, and those assumptions may very well make their architected solution a complete pile of garbage. Not to mention, how can a DS offer suggestions for solutions if they aren't deeply familiar with the raw data that's available?
To me, a good data scientist should, at bare minimum, have several skills.
* They should first and foremost (but not solely) be an in house expert in statistics and machine learning to know what can be done with data, and what can't be done with data. They should arrive with that knowledge. Engineers I think have a tendency to trivialize this, but true expertise in this domain comes only with years of experience.
* They should strive to find modeling solutions that are right for a particular business problem. If they seem to be only applying the hottest research regardless of the tradeoffs for the particular business problem, that's a red flag.
* Their focus should be on integrating themselves with the product/business as much as possible, and with the engineering team as much as possible. If they're expecting to be handed directives, that's a recipe for a ton of wasted time.
DS should never, ever be siloed into their own little DS world. They will be useless without a deeply intimate knowledge of the business goals, the needs of product, and the capabilities of the engineering team.
As they progress, they should become more and more "full-stack", otherwise they are stagnating.
Re: We don't need data scientists, we need data engineers
#69Preach! The data lifecycle is waaay overpopulated with Data Scientists who are not empowered or knowledgeable enough to work with product designers and engineers to do everything that empowers Data Science and ML. We need more Data Engineers involved at time zero in projects to help: 1. Plan out what data should be produced/captured by the product 2. Instrument systems to actually generate data consistently and effec…
[deleted]
Re: We don't need data scientists, we need data engineers
#70In many cases, what is needed isn't even more data engineers; it's data janitors.