Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Data engineering and software engineering are converging

clickhouse.com

Post reply on HN
Print view View original

31–40 of 68 posts

Re: Data engineering and software engineering are converging

#31

Post by getnormality »

sdairs wrote: post #12
getnormality wrote: post #9

It's not hard to do data engineering to the standards of software engineering, and many people do it already, provided that 1. You use a real programming language that supports all the abstractions software engineers rely on, not (just) SQL. 2. The data is not too big, so the feedback cycle is not too horrendously slow. #2 can't ever be fully solved, but testing a data pipeline on randomly subsampled data can help a…

In your experience, how are folks doing (1)? The post is talking about a framework to add e.g. type safety, schema-as-code, etc. over assets in data infra in a familiar way as to what is common with Postgres; I'm not familiar with much else out there for that?

Python, R, and Julia all have at least one package that defines a tabular data type. That means we can pass tables to functions, use them in classes, write tests for them, etc.

In all of these packages, the base tabular object you get is a local in-memory table. For manipulating remote SQL database tables, the best full-featured object API is provided by R's dbplyr package, IMHO.

I think Apache Spark, Apache Ibis, and some other big data packages can be configured to do this too, but IMHO their APIs are not nearly as helpful. For those who (understandably) don't want to use R and need an alternative to dbplyr, Apache Ibis is probably the best one to look at.

Re: Data engineering and software engineering are converging

#32

Post by vjvjvjvjghv »

craneca0 wrote: post #18
CalRobert wrote: post #5

Data engineering was software engineering from the very beginning. Then a bunch of business analysts who didn't know anything about writing software got jealous and said that if you knew SQL/DBT you were a data engineer. I've had to explain too many times that yes, indeed, I can set up a CI/CD pipeline or set up kafka or deploy Dagster on ECS, to the point where I think I need to change my title just to not be cheape…

yeah, i've seen large fortune 100 data and analytics orgs where the majority of folks with data engineering titles are uncomfortable with even the basics of git.

We have these at my company. They refuse to do any infrastructure work so you have to spoon feed the databases to them ready to go. It’s pretty annoying.

Re: Data engineering and software engineering are converging

#33

Post by kentm »

CalRobert wrote: post #5

Data engineering was software engineering from the very beginning. Then a bunch of business analysts who didn't know anything about writing software got jealous and said that if you knew SQL/DBT you were a data engineer. I've had to explain too many times that yes, indeed, I can set up a CI/CD pipeline or set up kafka or deploy Dagster on ECS, to the point where I think I need to change my title just to not be cheape…

Yep, I specifically asked my company to make sure my job title was not “data engineer” when working on data infrastructure, because there was a growing trend of using it to mean “can write some sql”.

Likewise, we had to steer HR away from “data engineer” because we got very mixed results with candidates.

Re: Data engineering and software engineering are converging

#34

Post by kentm »

mynameisash wrote: post #26

The comments here are... interesting, as they indicate a strong split between analysts and those engineers that can operationalize things. I see another dimension to it all. My title is senior data engineer at GAMMA/FAANG/whatever we're calling them. I have a CS degree and am firmly in the engineering. My passion, though, is in using software engineering and computer science principles to make very large-scale data p…

People who can utilize the tooling to process petabytes of data efficiently aren’t the ones that are catching flack. The people I’m thinking of basically run massive inefficient SQL queries and then throw their hands up when it runs slowly or gets an oom error. They don’t even know how to do an explain plan. And if you try to explain to them things like partitioning, indexes, sketches, etc then they are not able to comprehend and argue that it’s not their job to learn, and that it’s the “proper engineers” job to scale the processing.

Re: Data engineering and software engineering are converging

#35

Post by itsoktocry »

kentm wrote: post #33
CalRobert wrote: post #5

Data engineering was software engineering from the very beginning. Then a bunch of business analysts who didn't know anything about writing software got jealous and said that if you knew SQL/DBT you were a data engineer. I've had to explain too many times that yes, indeed, I can set up a CI/CD pipeline or set up kafka or deploy Dagster on ECS, to the point where I think I need to change my title just to not be cheape…

Yep, I specifically asked my company to make sure my job title was not “data engineer” when working on data infrastructure, because there was a growing trend of using it to mean “can write some sql”. Likewise, we had to steer HR away from “data engineer” because we got very mixed results with candidates.

Ironic, since "Data Engineers" are probably far more in demand right now than "Software Engineers".

Re: Data engineering and software engineering are converging

#36

Post by itsoktocry »

kentm wrote: post #34
mynameisash wrote: post #26

The comments here are... interesting, as they indicate a strong split between analysts and those engineers that can operationalize things. I see another dimension to it all. My title is senior data engineer at GAMMA/FAANG/whatever we're calling them. I have a CS degree and am firmly in the engineering. My passion, though, is in using software engineering and computer science principles to make very large-scale data p…

People who can utilize the tooling to process petabytes of data efficiently aren’t the ones that are catching flack. The people I’m thinking of basically run massive inefficient SQL queries and then throw their hands up when it runs slowly or gets an oom error. They don’t even know how to do an explain plan. And if you try to explain to them things like partitioning, indexes, sketches, etc then they are not able to c…

>And if you try to explain to them things like partitioning, indexes, sketches, etc then they are not able to comprehend and argue that it’s not their job to learn, and that it’s the “proper engineers” job to scale the processing.

Make up a person and attack him, literal strawman. You sound pleasant to work with.

Re: Data engineering and software engineering are converging

#37

Post by ludicity »

I think they've been fully converged in most strong practitioners for a long time.

There's a specific type of "data engineer" (quotes to indicate this is what they're called by the business, not to contest their legitimacy) that just writes lots of SQL, but they're usually a bad hire for businesses. They're approximately as expensive as what people call platform engineers, but platform engineers in the data space can usually do modelling as well.

When organizations split teams up by the most SWE-type DEs and the pure SQL ones, the latter all jockey to join the former team which causes a lot of drama too.

Re: Data engineering and software engineering are converging

#38

Post by kentm »

itsoktocry wrote: post #36
kentm wrote: post #34

Earlier quoted context omitted.

People who can utilize the tooling to process petabytes of data efficiently aren’t the ones that are catching flack. The people I’m thinking of basically run massive inefficient SQL queries and then throw their hands up when it runs slowly or gets an oom error. They don’t even know how to do an explain plan. And if you try to explain to them things like partitioning, indexes, sketches, etc then they are not able to c…

> And if you try to explain to them things like partitioning, indexes, sketches, etc then they are not able to comprehend and argue that it’s not their job to learn, and that it’s the “proper engineers” job to scale the processing. Make up a person and attack him, literal strawman. You sound pleasant to work with.

I’m referring to actual people I have worked and interacted with so no not made up.

They’re not engineers and shouldn’t have been labeled data engineers. They have some other value to the company, presumably, but trying to repackage them as data engineers does cause issues. That’s the topic of this thread.

Re: Data engineering and software engineering are converging

#39

Post by ozim »

itsoktocry wrote: post #35
kentm wrote: post #33

Earlier quoted context omitted.

Yep, I specifically asked my company to make sure my job title was not “data engineer” when working on data infrastructure, because there was a growing trend of using it to mean “can write some sql”. Likewise, we had to steer HR away from “data engineer” because we got very mixed results with candidates.

Ironic, since "Data Engineers" are probably far more in demand right now than "Software Engineers".

Only in places silly enough to believe software devs/enga cannot write SQL.

Re: Data engineering and software engineering are converging

#40

Post by hobs »

rawgabbit wrote: post #4
zurfer wrote: post #2

Maybe. On the one side you have something like dbt or Moosestack. On the other hand analytics and data pipelining is still a lot of no code tooling and I doubt it will go away. However I would love to learn more about how other people use coding agents to do DE tasks.

In Snowflake, I am now writing Python Stored Procedures that make REST API calls to things like Datadog REST API and dumping the JSON into a Snowflake table. I then unpack the JSON and transform it into a normalized table. So far it works reasonably well. This is possible using Snowflake's external access feature. https://docs.snowflake.com/en/developer-guide/external-netwo...

Right, but isn't snowflake like the most expensive way to run python? why not just host the python outside of snowflake and store it there?
Post reply on HN