There is no data engineering roadmap

It's probably worth noting that I was mostly talking about Data Engineering proper. I think "SQL is king" is (ime anyway?) comes from data analysts, who are usually not engineers are have more of a math of stats background. Data engineers do seem to write spark/pandas which is a different set of problems. I think a lot of the "big ball of sql mud" and brittleness just come from the fact that the people writing them aren't engineers and aren't following an engineering discipline. They're analysts trying to answer individual questions and find patterns to then pass off to ML engineering or data engineering teams to productionize. Or - they're lower-value pipelines that wouldn't be worth the time to ask an engineering team to try to prioritize them - so you either get a brittle thing that mostly works and need occasional love, or you do without it entirely.

Out of curiousity, what then makes a difference between a data engineer and an ETL developer?

> There is just so much potential variance depending on where you work and who you are, that it doesn't sit well with me to shoe-horn everyone into a pattern of "First learn Airflow, then learn BigQuery, then learn Spark" when you could have a successful career and never touch a single one of those tools.

I agree with this. My observation has been that people who start/stay in 1 place for too long end up knowing some tool(s) very well but not others. So surely then we should abstract away the tools and identify what's common about them (and SQL)?

When I produced classroom content for the graduate program that we ran at work, I used to focus more what's being done (ETL, data management, perf optimisation) and primitives like HTTP for REST, bits about DevOps, security, analysis. The idea being that if one is fluent in understanding the underlying problem, tooling becomes easier to pick up.

So I still think that there's a roadmap, and that roadmap is the underlying concepts that data engineering tries to solve. I've found that you can teach someone enough about:

- ETL & data management (incl modelling) - DBA - DevOps - Software (architecture and development)

to the point where they're not as good as the best people in those fields (if they're confining themselves so), but to the point where they become good at the parts of those fields that are relevant to data engineering. That's often my roadmap.

I don't disagree with you, and I believe you're probably teaching people many of the right things that will set them up to do well in the future. But...the tricky part about learning just concepts and not the skills is that they are quite abstract and its very hard for a junior to immediately see how they can applied in practise when getting their first job. They're also quite hard to demonstrate to an employer. Learning SQL gives you some tangible to do that can help you to learn something about many of those areas, maybe not too deep, but while also getting a hands-on skill that easier to demonstrate to an employer. The reason I think SQL is the right skill there is that its specific enough that its actually useful, but its generic enough that its widely applicable to many employers (rather than specialising on one specific tool, like say Spark).

For example, you might learn the theory behind internal combustion engines, but does that make you more attractive as a Mechanic's apprentice than a kid who's rebuilt his lawn mower's engine? Neither are 100% perfect, but the hands on experience might be easier to demonstrate that you can be effective on day 1, and you can learn some of that theory on the job. Idk if that's the best analogy, and I'm not saying it's 100% the case here.

I did a degree in Compture Science and learnt a lot of theory, its all super interesting and personally I still love understanding all of the theory and low level details behind software...but many of the DEs working in Enterprise don't have that background (or interest). And that's not a bad thing, they're just as competent data engineers as I am.

I guess it's a matter of what your end goal (and timeline) is. Some folks aren't looking to be a word-class DE expert, for some its just a job and DE is a booming space with above-average compensation. In that case, you're probably not inclined to sit in a class room and learn all the theory, you just want to land the job and get on with it. Nothing wrong with that at all, some of the best get-stuff-done engineers are like this. For others, even if they find the space fascinating and they want to be at the top, they might be more interested in one very specific hand-on area and they only need as much theory as they pick up day-to-day.

Anyway, I don't think your approach is wrong, and everything you're teaching is great for someone who is genuinely interested in the field of DE - but I also don't think its all absolutely necessary if you just want to get your first DE job.

RE. DE vs ETL developer; I think DE has become more of an umbrella term like "Software Engineer" which encompasses many different roles. An ETL developer probably falls under Data Engineering these days, its just a specific niche.

I think it will change or alleviate the tedium of the boring bits of exploring a dataset, figuring out if the dataset can join to another dataset in the way you thought of first, and writing a few of the dashboards.

Hopefully it means I can dig into the real problem (correctness, idempotency, and performance) rather than the tedious implementation.

Sounds like a good idea despite how many people diss certs on this forum. I've been trying to get a entry-level DevOps role for over a year and I can't even get a nibble—nothing! No replies, no interviews.

I feel like I'm a competent backend dev, even though a lot of times I feel imposter syndrome when I look at the things other developers are doing.