> All you need is l̶o̶v̶e̶ SQL I’ve ended up doing a lot of data engineering over the years, because I have a background in low-level search/databases systems coding and know text well. I have mixed feelings about the field precisely because it’s so SQL dominated. Data engineering can be unsatisfying if you thrive on writing reliable systems. There’s a whole lot of big-ball-of-inscrutable-SQL work out there which whe…
There is no data engineering roadmap
31–40 of 133 posts
Re: There is no data engineering roadmap
#32> All you need is l̶o̶v̶e̶ SQL I’ve ended up doing a lot of data engineering over the years, because I have a background in low-level search/databases systems coding and know text well. I have mixed feelings about the field precisely because it’s so SQL dominated. Data engineering can be unsatisfying if you thrive on writing reliable systems. There’s a whole lot of big-ball-of-inscrutable-SQL work out there which whe…
Frequently DE projects have no concept of Dev, Test, Staging and then Prod. Quite often its dev and then straight to Prod. SQL of course is to partly or fully blame for this.
My last job for an large insurance company, they happily set a best practices of 90% test coverage (which in itself ended up being an artificial, ritualistic goal) which is impossible to achieve with DE tasks.
Re: There is no data engineering roadmap
#33It’s overwhelming.
Re: There is no data engineering roadmap
#34Really enjoyed this article. Made me feel a tad inspired to write a similar one, "There is no Product Management roadmap", tailored for the many people looking to break into that discipline. Loooots of parallels.
Re: There is no data engineering roadmap
#35Good article but a bit short. Anyway two books I can recommend for data engineering: Designing data intensive applications-will give you a good overview of tools and the theories, algorithms, data structures behind it for different types of DB. High performance browser networking-throwing this in because it can help extend the book above outside of a data center to the last mile. Sometimes you can save yourself a bun…
Interesting. I work in the space currently (an ELT company) and I wouldn’t characterize the skill sets you mentioned as critical to a data team or a data driven organization. They are helpful, sure, but a lot of the problems that data teams face in my experience are around how to expose data to the organization in a meaningful, regular, digestible, actionable way. This means thinking about how to ingest and transform…
100% this. I basically work in this role in my organization. I'm not sure how typical my experience is compared with other people who call themselves 'data Engineers'. I'm usually put into the situation where I'm looking at how to capture a raw signal (out of an instrument, device or similar) and package it up as structured data and store it in some kind of meaningful format people can use.
Useful skills I think (other than SQL) are: -Some understanding of PLCs/SCADA/OPC communication world. -knowledge about Signal analysis/ DSP, Kalman Filters, Fourier Transforms etc. Especially useful when you need to store a raw signal as "data" of some kind. also increasingly useful when the underlying data is in some format like Video, Audio etc. -An Understanding of working with Time Series analysis can help with data transformation etc (downsampling a high frequency signal into lower frequency one for example).
Once you have captured the signal and stored it you move on to the second part how to make it available to end users in a meaningful and accessible way - this is where SQL knowledge and working with DBAs comes into it. In my experience it has become increasingly common you need to understand about cloud architecture etc as well. For example in my org more and more of our storage is moving to Azure so knowing how to navigate that stack and all the terms they use for things can help save a lot of time and frustration.
Lastly it can help if you are at least somewhat familiar with the BI tools the end users will be using to ultimately leverage the data.
Re: There is no data engineering roadmap
#36Data engineering which pays well happens exactly where databases fail and you need bespoke solutions.
Re: There is no data engineering roadmap
#37> All you need is l̶o̶v̶e̶ SQL I’ve ended up doing a lot of data engineering over the years, because I have a background in low-level search/databases systems coding and know text well. I have mixed feelings about the field precisely because it’s so SQL dominated. Data engineering can be unsatisfying if you thrive on writing reliable systems. There’s a whole lot of big-ball-of-inscrutable-SQL work out there which whe…
You made great points. Data Engineering cannot claim to support test-driven development to half the extent of the rest of software engineering. Frequently DE projects have no concept of Dev, Test, Staging and then Prod. Quite often its dev and then straight to Prod. SQL of course is to partly or fully blame for this. My last job for an large insurance company, they happily set a best practices of 90% test coverage (w…
And that then extends to why staging and even dev aren't very useful - the scale is part of the equation. Even just doing your dev loop, it's pretty easy to get something that's logically correct on your test sample but then when you run across the real dataset you find that 1 row out of every million is weird but you still gotta deal with it.
Re: There is no data engineering roadmap
#38Earlier quoted context omitted.
The problem is that if you have all the skills of a software engineer, lots of people would pay you more to do different things than they will to do "data engineering", which is often a lot about cobbling different tools together and doing a bunch of custom transformations and validations that are more tedious than creative.
A data engineer is as much a software engineer as a back-end engineer or front-end engineer and paid equivalently.
Re: There is no data engineering roadmap
#39Telling data engineers to learn sql because it's what's used in BoringCorp is a bit like telling front end developers in 2003 to learn VB because it's what most user interfaces are written in. Data engineering which pays well happens exactly where databases fail and you need bespoke solutions.