Engineers Shouldn’t Write ETL

ETL means "Extract, Transform, Load" https://en.wikipedia.org/wiki/Extract,_transform,_load

What’s funny to me is how many incompetent “thinkers” appear in meetings. Obviously, thought (even removed from implementation entirely) often has immense value. Eg, many people spent a lot of time thinking about arithmetic, linear algebra, floating point, compilers, and now I can go run whatever cool algorithm on my computer. But I continually seem to run into these people who seem borderline incompetent at anything but spewing out whatever pops into their head. Half is nonsense, one-quarter would be actively destructive if you tried to implement it, they always seem to know everything about everything but whenever it’s something you know really well you can tell that they are very confused, etc. when I meet these people now I just think “oh, you’re one of those guys who is good at saying a lot of things” and then move on. Oh well

At least in my Org Reports are pretty much an after thought left to the data engineers (like me) to "take this metric I've developed" and display it on the morning report.

Writing/updating a report is easiest part of my job it's the data that goes into building it that is hard translating the "simple metric I've developed" and getting it to run in a robust automated and sane fashion is the difficult part.

The complexities in my org are two fold.

Firstly the infrastructure people don't get data - at all. They speak PLC's and HMI's to them it's all OPC and magic A2A messaging takes care of everything. All data is time series to them and it all goes into an historian (which is basically a giant ring buffer i.e it gets flushed periodically) anything beyond that is past their level of expertise.

The data needs to be batched together the time series information has to be processed into "event frames" - this data was all part of this sequence of conveyor belt movements for example. Then you need to link it to related events etc and archive it in some kind of sane fashion so that in six months time if there is a product defect or something like that you can trace the entire series of event frames for that particular production batch.

Secondly the people the article calls "data scientists" (in my org these are Engineers - real ones of the Chem and Mech variety) don't know anything about databases or handling data they prototype their metrics in Matlab, Fortran, Excel and the like.

You really need someone to translate their code into something sane that can be automated. Engineers are not taught to code at all. I know I studied engineering at university Fortran is the lingua franca. Code is just a way of representing mathematics. Asking these people to do all the data processing pipeline is just not going to happen. It's not their job. They write the simulations and models they have the domain knowledge thats whats important for them to be worrying about.

I'm wondering if anybody lives in the hypothetical perfect world scenario the author writes about. I'm at one of the larger tech companies and it's inconceivable that something like this could exist (though the churn here is extremely high - a mature shop with longstanding membership could implement the hypothetical in some form). Everything sounds nice when dreaming it up in one's head, but discounts the reality of things. You can only lead a horse to water so many times before recognizing that they just will not drink the water themselves - some people actively refuse to implement solutions, no matter how convenient the building process might be. And then the more you burden them with things like SLAs, performance, etc., the more of shit show things become.

There are some forward-moving, solid "soft skill" analysts/data scientists that can make this happen. But by and large they shouldn't all be held to this standard. Maybe my standards/expectations have been soured too much and I'm too pessimistic, but as a whole they're just not cut out for this kind of stuff. Which is fine - being a "doer" is easy to begin with, and over time the more that you're able to automate as a data engineer, the more trivially easy ETL/everything else becomes.