Live data from Hacker News

There is no data engineering roadmap

alasdairb.com

31–40 of 133 posts

Re: There is no data engineering roadmap

#31
post #7

> All you need is l̶o̶v̶e̶ SQL I’ve ended up doing a lot of data engineering over the years, because I have a background in low-level search/databases systems coding and know text well. I have mixed feelings about the field precisely because it’s so SQL dominated. Data engineering can be unsatisfying if you thrive on writing reliable systems. There’s a whole lot of big-ball-of-inscrutable-SQL work out there which whe…

i dont understand what makes Pandas testable that is not also doable with the part of one's code that emits the SQL queries. assuming your test suite is capable of running with a database connection (which most are).

Re: There is no data engineering roadmap

#32
post #7

> All you need is l̶o̶v̶e̶ SQL I’ve ended up doing a lot of data engineering over the years, because I have a background in low-level search/databases systems coding and know text well. I have mixed feelings about the field precisely because it’s so SQL dominated. Data engineering can be unsatisfying if you thrive on writing reliable systems. There’s a whole lot of big-ball-of-inscrutable-SQL work out there which whe…

You made great points. Data Engineering cannot claim to support test-driven development to half the extent of the rest of software engineering.

Frequently DE projects have no concept of Dev, Test, Staging and then Prod. Quite often its dev and then straight to Prod. SQL of course is to partly or fully blame for this.

My last job for an large insurance company, they happily set a best practices of 90% test coverage (which in itself ended up being an artificial, ritualistic goal) which is impossible to achieve with DE tasks.

Re: There is no data engineering roadmap

#33
There’s long been framework fatigue in frontend JS work. I feel data engineering has a lot of that now. There’s different schedulers and orchestrators. Databases, ORMs, and wrappers. Cloud providers and hosting platforms. The list goes on.

It’s overwhelming.

Re: There is no data engineering roadmap

#34

Really enjoyed this article. Made me feel a tad inspired to write a similar one, "There is no Product Management roadmap", tailored for the many people looking to break into that discipline. Loooots of parallels.

i think you're probably onto something, what do you think the equivalent to SQL would be in product management?

Re: There is no data engineering roadmap

#35

Good article but a bit short. Anyway two books I can recommend for data engineering: Designing data intensive applications-will give you a good overview of tools and the theories, algorithms, data structures behind it for different types of DB. High performance browser networking-throwing this in because it can help extend the book above outside of a data center to the last mile. Sometimes you can save yourself a bun…

Interesting. I work in the space currently (an ELT company) and I wouldn’t characterize the skill sets you mentioned as critical to a data team or a data driven organization. They are helpful, sure, but a lot of the problems that data teams face in my experience are around how to expose data to the organization in a meaningful, regular, digestible, actionable way. This means thinking about how to ingest and transform…

>a lot of the problems that data teams face in my experience are around how to expose data to the organization in a meaningful, regular, digestible, actionable way. This means thinking about how to ingest and transform data and present it to end users for analysis (BI) in such a way that it’s not painful or arduous to consume it, and ensuring that it is clean and accurate

100% this. I basically work in this role in my organization. I'm not sure how typical my experience is compared with other people who call themselves 'data Engineers'. I'm usually put into the situation where I'm looking at how to capture a raw signal (out of an instrument, device or similar) and package it up as structured data and store it in some kind of meaningful format people can use.

Useful skills I think (other than SQL) are: -Some understanding of PLCs/SCADA/OPC communication world. -knowledge about Signal analysis/ DSP, Kalman Filters, Fourier Transforms etc. Especially useful when you need to store a raw signal as "data" of some kind. also increasingly useful when the underlying data is in some format like Video, Audio etc. -An Understanding of working with Time Series analysis can help with data transformation etc (downsampling a high frequency signal into lower frequency one for example).

Once you have captured the signal and stored it you move on to the second part how to make it available to end users in a meaningful and accessible way - this is where SQL knowledge and working with DBAs comes into it. In my experience it has become increasingly common you need to understand about cloud architecture etc as well. For example in my org more and more of our storage is moving to Azure so knowing how to navigate that stack and all the terms they use for things can help save a lot of time and frustration.

Lastly it can help if you are at least somewhat familiar with the BI tools the end users will be using to ultimately leverage the data.

Re: There is no data engineering roadmap

#36
Telling data engineers to learn sql because it's what's used in BoringCorp is a bit like telling front end developers in 2003 to learn VB because it's what most user interfaces are written in.

Data engineering which pays well happens exactly where databases fail and you need bespoke solutions.

Re: There is no data engineering roadmap

#37
post #32
post #7

> All you need is l̶o̶v̶e̶ SQL I’ve ended up doing a lot of data engineering over the years, because I have a background in low-level search/databases systems coding and know text well. I have mixed feelings about the field precisely because it’s so SQL dominated. Data engineering can be unsatisfying if you thrive on writing reliable systems. There’s a whole lot of big-ball-of-inscrutable-SQL work out there which whe…

You made great points. Data Engineering cannot claim to support test-driven development to half the extent of the rest of software engineering. Frequently DE projects have no concept of Dev, Test, Staging and then Prod. Quite often its dev and then straight to Prod. SQL of course is to partly or fully blame for this. My last job for an large insurance company, they happily set a best practices of 90% test coverage (w…

In my experience another big problem is that it's just expensive. For Spark jobs for instance, it's very common for it to run fine on the small test dataset on your laptop but then when you release it to run for 2 days on the prod dataset you end up having to do tuning there with pretty long turnaround times.

And that then extends to why staging and even dev aren't very useful - the scale is part of the equation. Even just doing your dev loop, it's pretty easy to get something that's logically correct on your test sample but then when you run across the real dataset you find that 1 row out of every million is weird but you still gotta deal with it.

Re: There is no data engineering roadmap

#38
post #30

Earlier quoted context omitted.

The problem is that if you have all the skills of a software engineer, lots of people would pay you more to do different things than they will to do "data engineering", which is often a lot about cobbling different tools together and doing a bunch of custom transformations and validations that are more tedious than creative.

A data engineer is as much a software engineer as a back-end engineer or front-end engineer and paid equivalently.

I would certainly prefer it if that were the case, but it largely is not, in my experience.

Re: There is no data engineering roadmap

#39
post #36

Telling data engineers to learn sql because it's what's used in BoringCorp is a bit like telling front end developers in 2003 to learn VB because it's what most user interfaces are written in. Data engineering which pays well happens exactly where databases fail and you need bespoke solutions.

They’re speaking to entry-level people, who will not be designing bespoke solutions. SQL is exactly the right recommendation.
Post reply on HN