We don't need data scientists, we need data engineers

Very true. I think I recently deliberated on this point about ideas being cheap because they can so deceptively sit in the realm of the ideal where everything is unconstrained and unchallenged. Grounding ideas and putting them to the test is where you begin to discover all the boundaries and tradeoffs and messy details that must be sorted through. Real work has a way of illuminating all those sticky messy points that must line up first for an idea to have a legging in the real world. Our imaginations are free to come up with all sorts of inconsistent and conflicting ideas that just never come to the foreground because of the centration on the beautiful perfect idea.

We have a sister company with many data scientists, and very few (actually I don't think that they ever hired any with the specific title) data engineers.

And, their production alleged "machine learning" (it's pretty much standard linear regression, but calling it ML is sexy) systems are slow motion train-wrecks. If the string and duct tape holds, then it works, but it's unfortunately continually breaking.

Hell, in Slack, I watch their data scientists continuously wrestle with how to actually make their Jupyter notebooks work in production.

Whereas my company has far more data engineers than data scientists. The plan from higher up the corporate food chain was always that we'd give them our data to do their data science voodoo on, but we ended up getting a few data scientists of our own for specific projects.

So, we focused on ensuring our data stream was reliable, consistent and sufficiently timely, for them to work on. But as soon as it hits their systems, it's a forest fire of hacks upon hacks, which inevitably break.

In the end, we had to send in our data engineers to stabilise their flagship "real time reporting" product that corporate was so amped about.

So yeah, I think that there's probably a happy ratio of data scientists to data engineers, of about, say, 1:5 or 1:10, because the maths generally scales O(1), it's the beauty of maths, but the actual engineering to get clean data delivered timely without breaking anything scales very differently indeed.

"When art critics get together they talk about Form and Structure and Meaning. When artists get together they talk about where you can buy cheap turpentine." — Picasso.

^Culture is good at romanticizing the "dreamer" as divorced from (& higher than) the "doer"/implementer. Picasso might protest. This post is helpful in inviting us examine this instinct/tradition.

But that good contrariness doesn't excuse us from being more thoughtful about the chain of deduction underlying the titular claim. First, many big ideas—Maxwell's equations of electromagnetism; that software should be Free as in both Beer and Software; that unruly Democracy could possibly be sometimes a (messily, weirdly) good way of organizing people—might naturally require more doers/thinkers to implement than just the mind (few minds?) who happened to crystallize it. Non-unitary stoichiometries for progress are the rule, not the exception. Nudging a culture is already a many-body problem, & because details matter, and details scale exponentially with levels of abstraction, a project's success can improve with the number of engineering minds adding leverage to advance it.

If we take the above earnestly—that making ideas useful usually requires more people than who happened to express an idea—then noticing that more job postings exist for implementory/engineering roles than for "science" roles actually says nearly nothing about whether we as a technological culture are out of balance with science vs engineering, up to how poorly we know about the typical ratio of implementers versus dramers.

It could be that there are plenty of good "scientific" ideas in circulation; maybe what separates us from progress is earnest implementation, reflected by empirical over-demand for engineers (as this post seems to mainly argue). The aggressive scaling laws for improving AI (along existing paradigms but broader compute) are tempting support for this conclusion.

But personally, I think it comes down to your position on this underlying question. Do you believe that fundamentally _better_ data paradigms--eg those that actually compute differently & more (super)humanly--will come from ideas already articulated in the conceptual universe? Or do you think that the key to smarter data science, if it exists, has yet to be invented and may little resemble the ideas dominantly in circulation?

If the latter, then we may most desperately need data _scientists_, in addition to engineers! In the sense that society would totally benefit from generating 1 new idea from “science,” even if 1000+ had been funded but not panned out. This is true on a global, pro-social sense, but also likely on an individual basis: if you are a thinker than surely partly what matters is what role could maximize your ∂impact/∂effort, and probabilistically, science of data is at least competitive with engineering if you think the future will look different than the present and needs to be invented.

I remember the same claims for ISL Clementine before IBM bought it and turned it into SPSS modeler.

The unspoken assumption in this article (and many of the replies in this thread) is that all statistical or machine learning problems are online problems, where data must be ingested, processed and analyzed continuously and indefinitely. For something like a SaaS app that's a no brainer because you have all of that data anyway, but the overwhelming amount of business analysis / optimization / modeling / prediction still revolves around one-off analyses and specific experiments with specific goals, for which all of this data engineering is overkill.

I've worked in teams that didn't get anything done because the data was a mess or there wasn't any, but I've also had the experience of spending so much time on the engineering side of things that we stopped asking what kind of things would actually be useful to know or learn or test. Computer scientists and engineers in general often have this attitude of "okay, let's take things one step at a time, first we focus on the data and get that 100% right" with a good amount of smugness about these data scientists and their messy, unorganized code, but consider that maybe they know something you don't.