We don't need data scientists, we need data engineers

I always felt that tech-focused data scientists should also be required to know how process data end-to-end; at minimum, from a SQL database to deployed model, but knowing how to collect & clean data is important too. It seems like the industry is trying fill the gap that was created by a glut of people without math/cs backgrounds going into 5-week data science courses who then need hand-holding when they get real jobs.

Data science & engineering should be treated as a single collection of skill-sets. Lacking ETL experience is a major deficit, considering how prevalent that kind of work is.

This might just be my personal biases coming through. I consider myself a "full-stack" data scientist & engineer. But because data scientists who can work on the backends are rare, I always end up doing the plumbing while other people do the fun analysis work.

I think companies that are data "science" heavy are going to be at huge disadvantage soon. Tools like Rekognition and Google AI APIs are making the model training & deployment aspect almost trivial. At some point, the only real work involved in this space will be the data "engineering."

Adopt a good Workflow tool, like apache airflow. Easiest is to rent a service from AWS https://aws.amazon.com/managed-workflows-for-apache-airflow/

As a data engineer I've made the same joke.

But the statement is also a bit like saying you can use plumbers to design and build a chemical refinement plant which also just moves chemicals from point A to point B. Or you can design a citywide sewer system with a bunch of plumbers.

There are many cleansing, refining, orchestration, dependency, data quality, governance and optimization problems to be solved and a wide variety of tools that have for whatever reason never grown into higher-level open source frameworks and are thus reimplemented in various forms in many places.

Data engineering (somewhat like software engineering actually) doesn't require much if any of the math and physics I took in engineering courses in college, but it does require rigorous systems thinking about how to design and build structures that withstand adverse conditions that are thought patterns common to other engineering practices, so I don't think it's a totally crazy title for the role.

I can't recommend the Data Engineer career enough for junior developers. It's how I started and what I pursued for 6 years (and I would love doing it again), and I feel like it gave me such an incredible foundation for future roles :

- Actually big data (so, not something you could grep...) will trigger your code in every possible way. You quickly learn that with trillions of input, the probabily to reach a bug is either 0% or 100%. In turn, you quickly learn to write good tests.

- You will learn distributed processing at a macro level, which in turn enlighten your thinking at a micro level. For example, even though the order of magnitudes are different, hitting data over network versus on disk is very much like hitting data on disk versus in cache. Except that when the difference ends up being in hours or days, you become much more sensible to that, so it's good training for your thoughts.

- Data engineering is full of product decisions. What's often called data "cleaning" is in fact one of the import product decisions made in a company, and a data engineer will be consistently exposed to his company product, which I think makes for great personal development

- Data engineering is fascinating. In adtech for example, logs of where ads are displayed are an unfiltered window on the rest of humanity, for the better or the worse. But it definitely expands your views on what the "average" person actually does on its computer (spoiler : it's mainly watching porn...), and challenges quite a bit what you might think is "normal"

- You'll be plumbing technologies from all over the web, which might or might not be good news for you.

So yeah, data engineering is great ! It's not harder than other specialties for developers, but imo, it's one of the fun ones !

>The actual process of collecting, aggregating, cleaning and verifying data is a hugely important skill, and not one I've really seen typical data scientists possess.

Then they are not scientists. They have a label "scientist" but lack of rigor of actual science.

I don't see why changing the label to "engineer" would suddenly make them have rigor.