Can't forget about SQL. It's also worth noting that designing any central data system without a SQL interface is probably a mistake. The people making it may be more comfortable with a REST interface or Python API, but data users will show up and demand that data be SQL-accessible. So now you get to hack it in, or help them develop their own shadow database that is poorly replicated from the main system.
There is no data engineering roadmap
11–20 of 133 posts
Re: There is no data engineering roadmap
#12I'm not a data engineer but what about dimensional modelling and stuff like the data warehouse toolkit? I would imagine that it is somewhat important?
It is important, but when the shift to cheap compute and storage hit an inflection point it became possible to build out wide tables that combine the characteristics of facts and dimensions, and real data modeling took a back seat. That possibility led to bias toward moving quickly with less emphasis on making the data model sustainable.
I'm seeing the pendulum start to swing the other way, where the complexity of these scrappy and loosely structured data models is hampering the ability to innovate, and even slowing down the business. The models are often inflexible and hard to maintain with hidden bugs and gotchas.
Re: There is no data engineering roadmap
#13The Data Engineering track seems to attract data analysts and non-engineers. I’m one of them, have worked with many others in my same bucket. What makes Data Engineers excel, I think, is to have all the skills of a software engineer + the ability to think like a data analyst.
SQL is definitely part of that journey, and more skills should be mastered if you’re going to do things like use Kafka, airflow or spark.
Re: There is no data engineering roadmap
#14Good article but a bit short. Anyway two books I can recommend for data engineering: Designing data intensive applications-will give you a good overview of tools and the theories, algorithms, data structures behind it for different types of DB. High performance browser networking-throwing this in because it can help extend the book above outside of a data center to the last mile. Sometimes you can save yourself a bun…
To add some additional notes - Designing Data Intensive Applications was a heavy read to dive into for me as a junior engineer. I took a step back after the first couple of chapters and read Understanding Distributed Systems[0] which was a fantastic primer.
Re: There is no data engineering roadmap
#15Good article but a bit short. Anyway two books I can recommend for data engineering: Designing data intensive applications-will give you a good overview of tools and the theories, algorithms, data structures behind it for different types of DB. High performance browser networking-throwing this in because it can help extend the book above outside of a data center to the last mile. Sometimes you can save yourself a bun…
Interesting. I work in the space currently (an ELT company) and I wouldn’t characterize the skill sets you mentioned as critical to a data team or a data driven organization. They are helpful, sure, but a lot of the problems that data teams face in my experience are around how to expose data to the organization in a meaningful, regular, digestible, actionable way. This means thinking about how to ingest and transform data and present it to end users for analysis (BI) in such a way that it’s not painful or arduous to consume it, and ensuring that it is clean and accurate. In other words, how well you move the data has always been a secondary point to how people can access and use said data.
Note that I’m not in a dedicated data engineering role, but have worked adjacent to it for the better part of a decade and have done a fair amount of data engineering in that time.
Re: There is no data engineering roadmap
#16The Data Engineering track seems to attract data analysts and non-engineers. I’m one of them, have worked with many others in my same bucket. What makes Data Engineers excel, I think, is to have all the skills of a software engineer + the ability to think like a data analyst. SQL is definitely part of that journey, and more skills should be mastered if you’re going to do things like use Kafka, airflow or spark.
for a moment there I thought you said "what makes data engineers is excel"
Re: There is no data engineering roadmap
#17> All you need is l̶o̶v̶e̶ SQL I’ve ended up doing a lot of data engineering over the years, because I have a background in low-level search/databases systems coding and know text well. I have mixed feelings about the field precisely because it’s so SQL dominated. Data engineering can be unsatisfying if you thrive on writing reliable systems. There’s a whole lot of big-ball-of-inscrutable-SQL work out there which whe…
Can’t speak for SQL _in general_ but in Postgres land pgTAP is wonderful. Invest in writing some factory functions, and you have the ability to write a very expressive test suite.
We use Postgres functions to provide a data interface that is thoroughly tested. This allows us to ensure that any migration does not break any contracts.
Re: There is no data engineering roadmap
#18One the most important thing about SQL is learn to read the output of EXPLAIN queries: it's the key to understand why your queries are not running as fast as you'd like.
Re: There is no data engineering roadmap
#19I'm not a data engineer but what about dimensional modelling and stuff like the data warehouse toolkit? I would imagine that it is somewhat important?
It is important, but when the shift to cheap compute and storage hit an inflection point it became possible to build out wide tables that combine the characteristics of facts and dimensions, and real data modeling took a back seat. That possibility led to bias toward moving quickly with less emphasis on making the data model sustainable. I'm seeing the pendulum start to swing the other way, where the complexity of th…
Interesting, this makes me think of a process I am going through, where I have a couple of very wide tables where I feel a need to build fact tables simply to get a better understanding of the data and the domain they are from. I have already stumbled a couple of times as the groupings was not as expected at all. Think this was the last push for me to build those fact tables once and for all, to get an overview and also to be able to add my own inferred groups to the data.
Re: There is no data engineering roadmap
#20I'm not a data engineer but what about dimensional modelling and stuff like the data warehouse toolkit? I would imagine that it is somewhat important?
A lot of it is dated but the first 3 chapters are still well worth a read for an aspiring DE.