There is no data engineering roadmap

I work as an (data) analyst, and I'll echo that many data engineers just end up in the engineering role after having worked as a traditional analyst.

You start doing the traditional analysis work, and then slowly get more involved on the tooling and pipeline side. Then you become to "go-to" guy on some specific tech, and slowly find yourself doing more and more engineering work, than writing traditional reports. You get offered more money to do the engineering work, due to engineering positions having higher salary caps than the analyst positions.

But it is a natural evolution. When you work with analysis, you obviously work very close with the data. You work close with the engineers (if your org / company has them), or become the engineer if there aren't any.

Software always involve data.

There is now a tendency for large companies to announce a slice of engineer's day to day to be a standalone position, and others will follow suit thinking they also need a dedicated position/team to handle such operation. And management training indoctrinates you that a team needs objective, measurable result, and sometimes a roadmap. Somewhere down the road, you realised that whatever your goal is, is 2 degrees detached from you business.

Data consistency, mean time to recovery, sprint velocity, DORA, SPACE, what else? It is as if one opens a ramen shop to find keeping the light bulbs on all the time the most important job. What about the ramen?

In micro service, there is this grains of sand anti-pattern where a service is too granular[1]. Zoom in a bit, you got left-pad as a library. Zoom out a bit, you got identity crisis of granular team. Don't forget to preach communication is important. How else are you going to coordinate all these services, I mean, teams, otherwise

[1] https://www.oreilly.com/content/microservices-antipatterns-a...

> It keeps them in the loop, give them context and perspective, and help them take better decisions

Other side of the coin: "I just spent 2 weeks optimising a daily process to be twice as fast" might well end up in me wondering why anyone would care about the 1 minute reduction in delivery time for daily cadence data. It might also result in me being stroppy about work I cannot get resourced that has obvious $ value.

But that is also a win for the business. Either I am wrong and am persuaded of the value, or I am not and I become more engaged with stakeholder prioritisation and we set up something more effective.

I view data engineering as a hybrid of SQL and programming. I'd go as far as saying that it's a mixture of various disciplines.

In the end it depends on what you'd be doing as a data engineer. My experience transitioning into the role/ecosystem in the last decade (I'm only 33) + hiring data analysts + scientists + engineers has left me with this impression:

Data engineering isn't so much an entry-level career, similar to data science if you don't have some core stat+math background. You could find a job that requires you to write some data pipelines, and it requires just SQL knowledge, and that's fine. However consider what similarity you have with an ETL developer (think Informatica, Oracle, SSIS).

I'm probably one of those gatekeepers because I do think there's some roadmap. I agree that it almost always starts with SQL. It's however important to acquire a broad toolkit though, because some people who've gone into data eng roles straight from uni often find themselves only knowing SQL or some bespoke tool that they're using at work, and end up struggling when they encounter the Sparks and glitters of the fiels.

Of the hiring processes that I've been involved with (say 20 candidates so far), I've found software engineers to transition better into the role, because most of the time candidates who come from that discipline have more experience debugging performance issues, which translate well into SQL optimisation and generally figuring one's way around tools.

Lastly, I'm self-taught. I'm professionally an accountant who started programming while in uni. I was fortunate to move into an analytics team at work and build my foundations from there. The analysts often had Math backgrounds, and I realised that I wouldn't cut it, so I focused on the tooling and giving them data fast and reliably. It was a lot of night-time and weekend tinkering to get to where I am.

One of the problems I’ve seen with folks in data engineering is an underdeveloped sense of data modeling and table design. And this article is an example of that. It heavily emphasized SQL, but didn’t mention anything about data modeling. If you aren’t designing your tables thoughtfully, no amount of SQL proficiency will save you.