31–40 of 50 posts
As a Data Engineer, I feel like a fixer. People think their data is incorrect and they ask me to fix it. It's similar to fixing someone's computer when they think its broken. Incredibly valuable but also somewhat unfulfilling in that you are rarely innovating/inventing like a SWE. There you are building tools that can change how people act/think. On the other hand, as a SWE/inventor you aren't desperately needed like…
Earlier quoted context omitted.
Yeah I’m thinking of changing my title back to software dev instead of DE - it’s sort of getting a bad rep.
How do you define "bad rep"?
I’ve always seen it as a Data Oriented Software Engineer but it seems that isn’t always the case, specially recently where I’ve been offered jobs that were basically analyst roles or BI roles.
In that talk, I outlined three (3) primary archetypes which still exist today:
- Data Engineering for Business Intelligence (often a rebranding of yesterday's "DBAs")
- Data Engineering for Data Science
- Data Engineering for Machine Learning (precursor for today's "ML Engineers")
Having worn all of these hats in the past, and many others, in different contexts, one of the key points I wanted to convey was that, in practice:
- The Role of a Data Engineer on a Team is Complementary and Often Defined By The Tasks That Others Don’t (Want To) Do Well
There are many guides, tutorials, and courses that like to talk about tools and technologies, and yet very few that get real about organizational structures and the dynamics that cause these archetypes to emerge, peoples' desires to rebrand themselves as "Data Engineers," "ML Engineers," or "(Data) Software Engineers" instead.
There was an interesting survey of ~600 "Data Engineers" by DataKitchen from 2021 ago where they had some very relatable, yet eye-opening findings:
- 97% report experiencing burnout in their day-to-day jobs.
- 70% say they are likely to leave their current company for another data engineering job in the next 12 months.
- 79% have considered leaving the industry entirely.
- 91% report frequently receiving requests for analytics with unrealistic or unreasonable expectations.
- 87% say they are blamed when things go wrong.
- 69% say their company’s data governance policies make their day-to-day job more difficult.
- 89% report frequent disruptions to work-life balance due to unplanned work.
(source: https://datakitchen.io/white_papers/2021-data-engineering-su...)
It's interesting to note, that when I was first called a DE - it was just software engineer in the data domain. As in writing full software, that happen to focus on data. Just 6 years ago I would be tinkering with PrestoDB code, looking at optimizing the scheduler and building Hadoop extensions. Between that and today the field swung to people who came from BI, with considerably less software engineering background.…
> software engineer in the data domain.
I was basically this for the past ten years. Maybe it was because I was working only in startups.
Outside of tech/startup orgs, "data engineer" at least I found, were SQL specialists. About six years ago, I went into healthcare, and discovered there were about 30 people across five teams that were data engineers. "Oh cool. My colleagues," I thought. Imagine my surprise when I found they only knew SQL, knew data modeling theory, and had basically no SDLC experience. At my present job, in a traditionally blue-collar industry, I took over a team with the only data engineer in the whole company. He, too, knew only SQL. I've had shove Python at him and get him working in SDLC.
I think these people though, are shrinking. Putting pressure on this from the other side, Python is a common skill with data analyst these days. Software engineers do the heavy lifting and good-enough data modeling, while data analyst do business-specific analysis and good-enough software development like writing DAGs with Dagster. Knowing SQL isn't enough to get by in the job market.
It's interesting to note, that when I was first called a DE - it was just software engineer in the data domain. As in writing full software, that happen to focus on data. Just 6 years ago I would be tinkering with PrestoDB code, looking at optimizing the scheduler and building Hadoop extensions. Between that and today the field swung to people who came from BI, with considerably less software engineering background.…
This lines up with my experience, and I've found it heavily depends on what industry you are in. > software engineer in the data domain. I was basically this for the past ten years. Maybe it was because I was working only in startups. Outside of tech/startup orgs, "data engineer" at least I found, were SQL specialists. About six years ago, I went into healthcare, and discovered there were about 30 people across five…
I personally haven’t met a lot of software devs who would call their data modeling capabilities ‘good enough’. It is a different way of thinking to go from building 3rd form normalized tables to denormalizing everything. Plus if a company is merging data from many systems, it makes it more complicated for the software developer.
Earlier quoted context omitted.
Yea same. Also, honestly feels like the way the field is progressing it will just be eaten up by an SWE role. Feel the same for ML engineer and many other specialized roles.
I don't think that SWEs will. The software and services are going to be getting advanced enough to just eliminate the need for a dedicated team to build ETL. People with relevant domain knowledge will have an easier time to deliver their work product, without the overhead of building phase. To get a reasonably good data platform - point-and-click ETL service, SAAS offering and the likes of Metabase - are already good…
Maybe I'm just of a different vintage but I'm just not a fan of these e-mail newsleters that seem to be the trend these days. I'd much rather follow a Github repo for this sort of thing.
Earlier quoted context omitted.
How do you define "bad rep"?
A lot of “data engineers” are former db analysts and such that don’t know much of anything technically outside of SQL and even that might be something they only are “certified” to know rather than actually good at. It’s basically becoming a title I’d associate with being low-skill. I used to be a “software engineer in data” and never call myself a data engineer because people would think I don’t know how to write/mai…
Just that I have had some recent experience with muti threaded java k8s services reading daily "streams" from Kafka.
When a SQL query would have solved it.
Earlier quoted context omitted.
This lines up with my experience, and I've found it heavily depends on what industry you are in. > software engineer in the data domain. I was basically this for the past ten years. Maybe it was because I was working only in startups. Outside of tech/startup orgs, "data engineer" at least I found, were SQL specialists. About six years ago, I went into healthcare, and discovered there were about 30 people across five…
I know plenty of people who only use sql. There is now a role called analytics engineer that primarily sql and often with transformation tools like dbt. I personally haven’t met a lot of software devs who would call their data modeling capabilities ‘good enough’. It is a different way of thinking to go from building 3rd form normalized tables to denormalizing everything. Plus if a company is merging data from many sy…