Data engineering and software engineering are converging • Hacker News Board

Re: Data engineering and software engineering are converging

#51

Post by orochimaaru » Sat, Aug 30, 2025, 4:17 AM UTC

rovr138 wrote: post #47

orochimaaru wrote: post #45

Earlier quoted context omitted.

If you’re using some variety of spark for your data engineering then scala is an option too. In general, choice of language isn’t important - again if you’re using spark your data frame structure schema defines that structure Python or not. Most folks confuse pandas with “data engineering”. It’s not. Most data engineering is spark.

in spark, doesn't pyspark and sql both still get translated to scala?

Yes. But with pyspark there is a Python gateway, the sql I think is translated natively in spark.

But when you create a dataframe in spark, that schema needs to be defined - or if it’s sql takes the form of the columns returned.

Use of Python can create hotspots with data transfers between spark and the Python gateway. Python UDFs are a common culprit.

Either way, my point is there are architectural and design points to your data solution that can cause many more problems than choice of language.

Re: Data engineering and software engineering are converging

#52

Post by mandeepj » Sat, Aug 30, 2025, 4:26 AM UTC

CalRobert wrote: post #5

Data engineering was software engineering from the very beginning. Then a bunch of business analysts who didn't know anything about writing software got jealous and said that if you knew SQL/DBT you were a data engineer. I've had to explain too many times that yes, indeed, I can set up a CI/CD pipeline or set up kafka or deploy Dagster on ECS, to the point where I think I need to change my title just to not be cheape…

> Data engineering was software engineering from the very beginning.

There it is! I found the post title was strange. Thanks for setting the record straight so succinctly.

Re: Data engineering and software engineering are converging

#53

Post by majormajor » Sat, Aug 30, 2025, 4:31 AM UTC

CalRobert wrote: post #5

Data engineering was software engineering from the very beginning. Then a bunch of business analysts who didn't know anything about writing software got jealous and said that if you knew SQL/DBT you were a data engineer. I've had to explain too many times that yes, indeed, I can set up a CI/CD pipeline or set up kafka or deploy Dagster on ECS, to the point where I think I need to change my title just to not be cheape…

"Data Engineering" being considered a different role from "regular" SWE predates DBT by... at least one decade? If not two? Probably folks working with Hadoop vs RDMS DBA jobs.

Re: Data engineering and software engineering are converging

#54

Post by majormajor » Sat, Aug 30, 2025, 4:33 AM UTC

itsoktocry wrote: post #35

kentm wrote: post #33

Earlier quoted context omitted.

Yep, I specifically asked my company to make sure my job title was not “data engineer” when working on data infrastructure, because there was a growing trend of using it to mean “can write some sql”. Likewise, we had to steer HR away from “data engineer” because we got very mixed results with candidates.

Ironic, since "Data Engineers" are probably far more in demand right now than "Software Engineers".

The "write SQL for ETLs all day" job is a risky one right now since LLMs really lower the barrier for dealing with gnarly SQL. So it's still not a bad time to have your resume be as clear as possible that you're the "deals with complex distributed systems" SWE type instead.

Re: Data engineering and software engineering are converging

#55

Post by teleforce » Sat, Aug 30, 2025, 4:41 AM UTC

For the foundation on data engineering I'd recommend this book by Joe Reis and Matt Housley. They did a good job on providing the framework that includes data engineering lifecycle, software engineering, data management, data architecture, etc. You can check the proposed framework here [1],[2].

[1] Fundamentals of Data Engineering:

https://www.oreilly.com/library/view/fundamentals-of-data/97...

[2] Fundamentals of Data Engineering Review:

https://maninekkalapudi.medium.com/fundamentals-of-data-engi...

Re: Data engineering and software engineering are converging

#56

Post by mrugge » Sat, Aug 30, 2025, 5:03 AM UTC

majormajor wrote: post #54

itsoktocry wrote: post #35

Earlier quoted context omitted.

Ironic, since "Data Engineers" are probably far more in demand right now than "Software Engineers".

The "write SQL for ETLs all day" job is a risky one right now since LLMs really lower the barrier for dealing with gnarly SQL. So it's still not a bad time to have your resume be as clear as possible that you're the "deals with complex distributed systems" SWE type instead.

That's more of an analytics engineer role. LLMs lower the barrier to entry, but popular SQL queries are about correctness and flexibility and this often requires deep understanding and ownership of each filter and window function. This lower barrier can quickly can turn into enough rope to hang yourself.

Re: Data engineering and software engineering are converging

#57

Post by mrugge » Sat, Aug 30, 2025, 5:13 AM UTC

This split between the main app stack and the data engineering / analytics stack is a time-tested architectural pattern. Has clickhouse changed the game so much that it is no longer helpful to have these purpose-built stacks? With modern coding agents being able to write more faster it might be good to explore more separation and purpose-built stacks and less convergence.

Re: Data engineering and software engineering are converging

#58

Post by mrugge » Sat, Aug 30, 2025, 5:20 AM UTC

CalRobert wrote: post #5

Data engineering was software engineering from the very beginning. Then a bunch of business analysts who didn't know anything about writing software got jealous and said that if you knew SQL/DBT you were a data engineer. I've had to explain too many times that yes, indeed, I can set up a CI/CD pipeline or set up kafka or deploy Dagster on ECS, to the point where I think I need to change my title just to not be cheape…

Titles in software engineering have never mattered less than they do today. Energy worrying about titles or jealosy over specific tech ownership is best channeled into focus on customer, on problem to solve and on finding the best way to solve it as a team.

Re: Data engineering and software engineering are converging

#59

Post by mrugge » Sat, Aug 30, 2025, 5:34 AM UTC

antupis wrote: post #50

greekorich wrote: post #46

Earlier quoted context omitted.

I've been a professional java dev for a decade. I've written a little python, clojure, lots of JS/TS/Node. SQL is the most beautiful, expressive, get stuff done language I've used. It is perfect for whatever data engineering is defined as.

SQL is beautiful when it works but when it doesn’t you end up with some abomination eg if you need some kind dynamic query.

The two most helpful things with SQL are (1) always use set-based operations (never a cursor) and (2) break up your queries into smallest possible reusable chunks (CTEs). Sprinkle in tests to taste. Without some discipline SQL can get out of hand. This is what made dbt popular.

Re: Data engineering and software engineering are converging

#60

Post by physicles » Sat, Aug 30, 2025, 7:27 AM UTC

sbrother wrote: post #49

jochem9 wrote: post #44

One thing that I don't see mentioned but that does bug me: data engineers often use a lot of Python and SQL, even the ones that have heavily adopted software engineering best practices. Yet both languages are not great for this. Python is dynamically typed, which you can patch a bit with type hints, but it's still easy to go to production with incompatible types, leading to panics in prod. It's uncompiled nature also…

When I was most recently at Google (2021-ish) my team owned a bunch of SQL Pipelines that had fairly effective SQL tests. Not my favorite thing to work on, but it was a productive way to transform data. There are lots of open source versions of the same idea, but I have yet to see them accompanied with ergonomic testing. Any recommendations or pointers to open source SQL testing frameworks?

Could you describe what made those tests effective? I just wrote some tools to write concise tests for some analytics queries, and some principles I stumbled on are:

- input data should be pseudorandom, so the chance of a test being “accidentally correct” is minimized

- you need a way to verify only part of the result set. Or, at the very least, a way to write tests so that if you add a column to the result set, your test doesn’t automatically break

In addition, I added CSV exports so you can verify the results by hand, and hot-reload for queries with CTEs — if you change a .sql file then it will immediately rerun each CTE incrementally and show you which ones’ output changed.