Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Data engineering and software engineering are converging

clickhouse.com

Post reply on HN
Print view View original

61–68 of 68 posts

Re: Data engineering and software engineering are converging

#61

Post by CalRobert »

kentm wrote: post #34
mynameisash wrote: post #26

The comments here are... interesting, as they indicate a strong split between analysts and those engineers that can operationalize things. I see another dimension to it all. My title is senior data engineer at GAMMA/FAANG/whatever we're calling them. I have a CS degree and am firmly in the engineering. My passion, though, is in using software engineering and computer science principles to make very large-scale data p…

People who can utilize the tooling to process petabytes of data efficiently aren’t the ones that are catching flack. The people I’m thinking of basically run massive inefficient SQL queries and then throw their hands up when it runs slowly or gets an oom error. They don’t even know how to do an explain plan. And if you try to explain to them things like partitioning, indexes, sketches, etc then they are not able to c…

My boss at a large company years ago wrote a query for daily stats and then proceeded to run it on the entire event history every day for the life of the company just to get DAU, etc. The solution was to just keep paying more for redshift until the bill was a few million a year. Suggestions to fix his crap were met with disdain.

That job taught me a lot.

Re: Data engineering and software engineering are converging

#62

Post by snthpy »

majormajor wrote: post #53
CalRobert wrote: post #5

Data engineering was software engineering from the very beginning. Then a bunch of business analysts who didn't know anything about writing software got jealous and said that if you knew SQL/DBT you were a data engineer. I've had to explain too many times that yes, indeed, I can set up a CI/CD pipeline or set up kafka or deploy Dagster on ECS, to the point where I think I need to change my title just to not be cheape…

"Data Engineering" being considered a different role from "regular" SWE predates DBT by... at least one decade? If not two? Probably folks working with Hadoop vs RDMS DBA jobs.

In work yes, but as a title? I only started seeing it called that around dbt origin.

Re: Data engineering and software engineering are converging

#63

Post by CyberDildonics »

itsoktocry wrote: post #35
kentm wrote: post #33

Earlier quoted context omitted.

Yep, I specifically asked my company to make sure my job title was not “data engineer” when working on data infrastructure, because there was a growing trend of using it to mean “can write some sql”. Likewise, we had to steer HR away from “data engineer” because we got very mixed results with candidates.

Ironic, since "Data Engineers" are probably far more in demand right now than "Software Engineers".

Why would a software engineer not be able to do both roles?

Re: Data engineering and software engineering are converging

#64

Post by food4u »

mynameisash wrote: post #26

The comments here are... interesting, as they indicate a strong split between analysts and those engineers that can operationalize things. I see another dimension to it all. My title is senior data engineer at GAMMA/FAANG/whatever we're calling them. I have a CS degree and am firmly in the engineering. My passion, though, is in using software engineering and computer science principles to make very large-scale data p…

[dead]

Re: Data engineering and software engineering are converging

#65

Post by yndoendo »

giantg2 wrote: post #3

I've never really seen the distinction between data and software engineering. It's more like front-end vs backend. If you're a data engineer and it's all no code tooling, then you're just an analyst or something.

The difference in titles is more or less where most of the time is spent. Developer could be doing front-end, back-end, embedded, high-performance computing, system, game, data analysis, or any other niche work. All of those have different design, tooling, and ways of thinking that you gain through actually doing.

I been in interviews where after reading my resume they say oh your an embedded developer. Another said a front-end, no a back-end, no a system developer, and other desktop developer. Reality, I did all of those to get the job done and create a viable product.

Re: Data engineering and software engineering are converging

#66

Post by omgwtfbyobbq »

CalRobert wrote: post #5

Data engineering was software engineering from the very beginning. Then a bunch of business analysts who didn't know anything about writing software got jealous and said that if you knew SQL/DBT you were a data engineer. I've had to explain too many times that yes, indeed, I can set up a CI/CD pipeline or set up kafka or deploy Dagster on ECS, to the point where I think I need to change my title just to not be cheape…

Part of the problem is that a BA/BSA who writes Python, SQL, etc... as part of their day to day work will get lumped with those who don't and their salary doesn't reflect their skills and work product.

Obviously the same can apply in any given title, and does with data engineers like you pointed out, but it's not as simple as just title inflation.

Re: Data engineering and software engineering are converging

#67

Post by omgwtfbyobbq »

CyberDildonics wrote: post #63
itsoktocry wrote: post #35

Earlier quoted context omitted.

Ironic, since "Data Engineers" are probably far more in demand right now than "Software Engineers".

Why would a software engineer not be able to do both roles?

They can, but it takes time away from software engineering.

Along the same reason, that's why there are DBAs, dev-ops engineers, etc...

Re: Data engineering and software engineering are converging

#68

Post by omgwtfbyobbq »

antupis wrote: post #50
greekorich wrote: post #46

Earlier quoted context omitted.

I've been a professional java dev for a decade. I've written a little python, clojure, lots of JS/TS/Node. SQL is the most beautiful, expressive, get stuff done language I've used. It is perfect for whatever data engineering is defined as.

SQL is beautiful when it works but when it doesn’t you end up with some abomination eg if you need some kind dynamic query.

Can you share more detail? Every pain point I've seen can be handled by a more suitable/performant/flexible DB and/or software that makes working with SQL less painful.
Post reply on HN