Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Data Engineering Design Patterns

dedp.online

Post reply on HN
Print view View original

21–30 of 50 posts

Re: Data Engineering Design Patterns

#21

Post by fifilura »

ed_elliott_asc wrote: post #19
JAlexoid wrote: post #11

It's interesting to note, that when I was first called a DE - it was just software engineer in the data domain. As in writing full software, that happen to focus on data. Just 6 years ago I would be tinkering with PrestoDB code, looking at optimizing the scheduler and building Hadoop extensions. Between that and today the field swung to people who came from BI, with considerably less software engineering background.…

Yeah I’m thinking of changing my title back to software dev instead of DE - it’s sort of getting a bad rep.

How do you define "bad rep"?

Re: Data Engineering Design Patterns

#24

Post by opportune »

fifilura wrote: post #21
ed_elliott_asc wrote: post #19

Earlier quoted context omitted.

Yeah I’m thinking of changing my title back to software dev instead of DE - it’s sort of getting a bad rep.

How do you define "bad rep"?

A lot of “data engineers” are former db analysts and such that don’t know much of anything technically outside of SQL and even that might be something they only are “certified” to know rather than actually good at.

It’s basically becoming a title I’d associate with being low-skill. I used to be a “software engineer in data” and never call myself a data engineer because people would think I don’t know how to write/maintain production services, just write ETL pipelines

Re: Data Engineering Design Patterns

#25

Post by icedchai »

qsort wrote: post #14
JAlexoid wrote: post #11

It's interesting to note, that when I was first called a DE - it was just software engineer in the data domain. As in writing full software, that happen to focus on data. Just 6 years ago I would be tinkering with PrestoDB code, looking at optimizing the scheduler and building Hadoop extensions. Between that and today the field swung to people who came from BI, with considerably less software engineering background.…

The BI world is honestly kind of weird. You have people who are at the intersection of "understands databases, the relational model, query optimization etc. at the level of a very senior SWE" ∩ "needs to be told how git works in the year of our lord 2023".

Don't forget the person who thinks they really have "big data" and needs all this massive infrastructure. Eventually, one discovers it's a few gigs of CSV files that fits in RAM on a laptop.

Re: Data Engineering Design Patterns

#26

Post by stvltvs »

JAlexoid wrote: post #15
lysecret wrote: post #13

Earlier quoted context omitted.

Yea same. Also, honestly feels like the way the field is progressing it will just be eaten up by an SWE role. Feel the same for ML engineer and many other specialized roles.

I don't think that SWEs will. The software and services are going to be getting advanced enough to just eliminate the need for a dedicated team to build ETL. People with relevant domain knowledge will have an easier time to deliver their work product, without the overhead of building phase. To get a reasonably good data platform - point-and-click ETL service, SAAS offering and the likes of Metabase - are already good…

My experience is that many established companies are still struggling to get adequate operational reporting. Data engineers are still helpful to move the data necessary to make that happen. DS and ML become useful later once there's a more mature data culture and infrastructure. Otherwise you have analysts spending most of their time doing data engineering so they have something to analyze.

Re: Data Engineering Design Patterns

#27

Post by alexyz12 »

As a Data Engineer, I feel like a fixer. People think their data is incorrect and they ask me to fix it. It's similar to fixing someone's computer when they think its broken.

Incredibly valuable but also somewhat unfulfilling in that you are rarely innovating/inventing like a SWE. There you are building tools that can change how people act/think. On the other hand, as a SWE/inventor you aren't desperately needed like you are as a DE. The needing can be nice...

Re: Data Engineering Design Patterns

#28

Post by Centigonal »

alexyz12 wrote: post #27

As a Data Engineer, I feel like a fixer. People think their data is incorrect and they ask me to fix it. It's similar to fixing someone's computer when they think its broken. Incredibly valuable but also somewhat unfulfilling in that you are rarely innovating/inventing like a SWE. There you are building tools that can change how people act/think. On the other hand, as a SWE/inventor you aren't desperately needed like…

I work in data, but I've seen a lot of comments from SWEs who are tired of building the same CRUD web app over and over again. The grass is always greener?

Re: Data Engineering Design Patterns

#29

Post by rockwotj »

icedchai wrote: post #25
qsort wrote: post #14

Earlier quoted context omitted.

The BI world is honestly kind of weird. You have people who are at the intersection of "understands databases, the relational model, query optimization etc. at the level of a very senior SWE" ∩ "needs to be told how git works in the year of our lord 2023".

Don't forget the person who thinks they really have "big data" and needs all this massive infrastructure. Eventually, one discovers it's a few gigs of CSV files that fits in RAM on a laptop.

I feel like duckdb has been making waves here helping people realize you don’t need snowflake/bigquery/etc for all ypur datasets, but you can still get the nice feature set of those systems

Re: Data Engineering Design Patterns

#30

Post by edmundsauto »

alexyz12 wrote: post #27

As a Data Engineer, I feel like a fixer. People think their data is incorrect and they ask me to fix it. It's similar to fixing someone's computer when they think its broken. Incredibly valuable but also somewhat unfulfilling in that you are rarely innovating/inventing like a SWE. There you are building tools that can change how people act/think. On the other hand, as a SWE/inventor you aren't desperately needed like…

Also a DE - the “data as a service” model is broken. Data engineers need to be upstream producing insights because they have the best feel for the warts.

Companies that treat DE as IT (“service model”) do not have the right perspective and will not hire the kind of DE who are difference makers.

Data teams should be partners.

Post reply on HN