Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

There is no data engineering roadmap

alasdairb.com

Post reply on HN
Print view View original

21–30 of 133 posts

Re: There is no data engineering roadmap

#21

Post by wokwokwok »

I think there's a fair caveat here:

In a context free situation, sure, just learn SQL. It's the workhorse, and you'll definitely use it.

If you have to start somewhere, start there.

...buuuut, obviously look at job ads.

If all the junior jobs say "experience with SnowPanda" or "Working knowledge of FancyCloudPlatform" or whatever the name of the latest hotness is, you'd be daft not to invest some time and effort into whatever everyone is actually asking for to.

You don't need someone to tell you what those things are; literally look at the bullet points on job ads... but don't ignore them.

Re: There is no data engineering roadmap

#22

Post by xmprt »

rectang wrote: post #7

> All you need is l̶o̶v̶e̶ SQL I’ve ended up doing a lot of data engineering over the years, because I have a background in low-level search/databases systems coding and know text well. I have mixed feelings about the field precisely because it’s so SQL dominated. Data engineering can be unsatisfying if you thrive on writing reliable systems. There’s a whole lot of big-ball-of-inscrutable-SQL work out there which whe…

> Data engineering can be unsatisfying if you thrive on writing reliable systems

This is what I needed for me to stop considering data engineering. Not that I was thinking about it a lot. I feel very happy with my current job but the thought of the code I write being so brittle as to paralyze me from making any changes terrifies me.

Re: There is no data engineering roadmap

#23

Post by Jarwain »

SOLAR_FIELDS wrote: post #15
chrismarlow9 wrote: post #8

Good article but a bit short. Anyway two books I can recommend for data engineering: Designing data intensive applications-will give you a good overview of tools and the theories, algorithms, data structures behind it for different types of DB. High performance browser networking-throwing this in because it can help extend the book above outside of a data center to the last mile. Sometimes you can save yourself a bun…

Interesting. I work in the space currently (an ELT company) and I wouldn’t characterize the skill sets you mentioned as critical to a data team or a data driven organization. They are helpful, sure, but a lot of the problems that data teams face in my experience are around how to expose data to the organization in a meaningful, regular, digestible, actionable way. This means thinking about how to ingest and transform…

This kinda sounds like higher level concerns where the books cover lower level problems?

Re: There is no data engineering roadmap

#24

Post by jeremyjh »

woleium wrote: post #16
iblaine wrote: post #13

The Data Engineering track seems to attract data analysts and non-engineers. I’m one of them, have worked with many others in my same bucket. What makes Data Engineers excel, I think, is to have all the skills of a software engineer + the ability to think like a data analyst. SQL is definitely part of that journey, and more skills should be mastered if you’re going to do things like use Kafka, airflow or spark.

for a moment there I thought you said "what makes data engineers is excel"

do not speaks its name

Re: There is no data engineering roadmap

#26

Post by sanderjd »

iblaine wrote: post #13

The Data Engineering track seems to attract data analysts and non-engineers. I’m one of them, have worked with many others in my same bucket. What makes Data Engineers excel, I think, is to have all the skills of a software engineer + the ability to think like a data analyst. SQL is definitely part of that journey, and more skills should be mastered if you’re going to do things like use Kafka, airflow or spark.

The problem is that if you have all the skills of a software engineer, lots of people would pay you more to do different things than they will to do "data engineering", which is often a lot about cobbling different tools together and doing a bunch of custom transformations and validations that are more tedious than creative.

Re: There is no data engineering roadmap

#27

Post by chrismarlow9 »

Jarwain wrote: post #23
SOLAR_FIELDS wrote: post #15

Earlier quoted context omitted.

Interesting. I work in the space currently (an ELT company) and I wouldn’t characterize the skill sets you mentioned as critical to a data team or a data driven organization. They are helpful, sure, but a lot of the problems that data teams face in my experience are around how to expose data to the organization in a meaningful, regular, digestible, actionable way. This means thinking about how to ingest and transform…

This kinda sounds like higher level concerns where the books cover lower level problems?

I think it's just two sides of the same coin. There's definitely an art to making it consumable to end users. It's even an engineering challenge to open these data lakes to end users without killing the database (think of a dev trying to show load average on a graph for all running servers over last 3 years and there's 5000 servers). You can't really blame them and realistically they might have a project where they need that data. And then there's data crazy people who make 400 graphs of various metrics and only 20 get actively used. It can lead you down red herring paths and all kinds of chaos as an SRE, and also blow the budget.

Anyway, I understand what they are saying and unfortunately I have no book recommendations to solve these problems. In larger companies it can be much more of a challenge than actually ingesting data, especially in the observability space.

Re: There is no data engineering roadmap

#28

Post by conkeisterdoor »

SOLAR_FIELDS wrote: post #15
chrismarlow9 wrote: post #8

Good article but a bit short. Anyway two books I can recommend for data engineering: Designing data intensive applications-will give you a good overview of tools and the theories, algorithms, data structures behind it for different types of DB. High performance browser networking-throwing this in because it can help extend the book above outside of a data center to the last mile. Sometimes you can save yourself a bun…

Interesting. I work in the space currently (an ELT company) and I wouldn’t characterize the skill sets you mentioned as critical to a data team or a data driven organization. They are helpful, sure, but a lot of the problems that data teams face in my experience are around how to expose data to the organization in a meaningful, regular, digestible, actionable way. This means thinking about how to ingest and transform…

I currently work in a Data Engineering role and agree with your assessment. At the business I work for, there was no BI or data management processes automation until I joined and wrote all the pipelines (in Python) and queries (in T-SQL). Being comfortable with Python and pandas/arrow (or some other language or ETL/ELT platform), SQL, and working with REST APIs is all usually necessary or helpful, but SQL is ubiquitous. And the most challenging problems for the data mgmt team I'm on involve:

(1) getting datasets from historically siloed data management teams to join with each other cleanly and programmatically — which in my case is a political issue, not a technical one

(2) determining the best way to replace old tabular (Excel) reporting with modern interactive (Tableau) dashboards that refresh from a database — which in my case usually means getting a better understanding of many cross-team business processes, understanding how data management is involved in them, and how data is being used to make decisions and take action

(3) determining how to structure and present this data to them with the least possible friction, and in a way that's the most useful and impactful to them

This is all to say that DE roles seem to vary wildly depending on the domain you work in and the kind of business you work for. But at the end of the day the purpose of the role is to make clean and accurate data available to end users with the least friction and in the most impactful way. And most kinds of data businesses use can be represented by, transformed with, and interacted with using SQL.

There's tons of SQL resources, but I'd love to find some books that cover these deeper issues of politics and business process discovery from a data engineering perspective.

Re: There is no data engineering roadmap

#29

Post by sroussey »

conkeisterdoor wrote: post #28
SOLAR_FIELDS wrote: post #15

Earlier quoted context omitted.

Interesting. I work in the space currently (an ELT company) and I wouldn’t characterize the skill sets you mentioned as critical to a data team or a data driven organization. They are helpful, sure, but a lot of the problems that data teams face in my experience are around how to expose data to the organization in a meaningful, regular, digestible, actionable way. This means thinking about how to ingest and transform…

I currently work in a Data Engineering role and agree with your assessment. At the business I work for, there was no BI or data management processes automation until I joined and wrote all the pipelines (in Python) and queries (in T-SQL). Being comfortable with Python and pandas/arrow (or some other language or ETL/ELT platform), SQL, and working with REST APIs is all usually necessary or helpful, but SQL is ubiquito…

I wrote Excel macros that connected to a VAX VMS DB2 to get updated data so that management could just open the same excel files as they always did. This was the early/mid 1990s

Re: There is no data engineering roadmap

#30

Post by _bk90 »

sanderjd wrote: post #26
iblaine wrote: post #13

The Data Engineering track seems to attract data analysts and non-engineers. I’m one of them, have worked with many others in my same bucket. What makes Data Engineers excel, I think, is to have all the skills of a software engineer + the ability to think like a data analyst. SQL is definitely part of that journey, and more skills should be mastered if you’re going to do things like use Kafka, airflow or spark.

The problem is that if you have all the skills of a software engineer, lots of people would pay you more to do different things than they will to do "data engineering", which is often a lot about cobbling different tools together and doing a bunch of custom transformations and validations that are more tedious than creative.

A data engineer is as much a software engineer as a back-end engineer or front-end engineer and paid equivalently.
Post reply on HN