Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Data engineering and software engineering are converging

clickhouse.com

Post reply on HN
Print view View original

11–20 of 68 posts

Re: Data engineering and software engineering are converging

#11

Post by zamalek »

RobinL wrote: post #8
zamalek wrote: post #6

One things have seen through my more recent exposure to experienced data engineers is the lack of repeatability rigor (CI/CD, IaC, etc.). There's a lot of doing things in notebooks and calling that production-ready. Databricks has git (GitHub only from what I can tell) integration, but that's just checking out and directly committing to trunk, if it's in git then we have SDLC right, right? It's fucking nuts. Anyone h…

I think this may be a databricks thing? From what I've seen there's a gap between data engineers forced to use databricks and everyone else. From what I've seen, at least how it's used in practice, databricks seems to result in a mess of notebooks with poor dependency and version management.

Interesting, databricks has been my first exposure to DE at scale and it does seem to solve many problems (even though it sounds like it's causing some). So what does everyone else do? Run spark etc. themselves?

Re: Data engineering and software engineering are converging

#12

Post by sdairs »

getnormality wrote: post #9

It's not hard to do data engineering to the standards of software engineering, and many people do it already, provided that 1. You use a real programming language that supports all the abstractions software engineers rely on, not (just) SQL. 2. The data is not too big, so the feedback cycle is not too horrendously slow. #2 can't ever be fully solved, but testing a data pipeline on randomly subsampled data can help a…

In your experience, how are folks doing (1)? The post is talking about a framework to add e.g. type safety, schema-as-code, etc. over assets in data infra in a familiar way as to what is common with Postgres; I'm not familiar with much else out there for that?

Re: Data engineering and software engineering are converging

#14

Post by sdairs »

zamalek wrote: post #11
RobinL wrote: post #8

Earlier quoted context omitted.

I think this may be a databricks thing? From what I've seen there's a gap between data engineers forced to use databricks and everyone else. From what I've seen, at least how it's used in practice, databricks seems to result in a mess of notebooks with poor dependency and version management.

Interesting, databricks has been my first exposure to DE at scale and it does seem to solve many problems (even though it sounds like it's causing some). So what does everyone else do? Run spark etc. themselves?

tbh I see just as much notebook-hell outside of dbx, it's certainly not contained to just them. There's some folks doing good SDLC with Spark jobs in java/scala, but I've never found it to be overly common, I see "dump it on the shared drive" equally as much lol. IME data has always been a bit behind in this area

personally you couldn't pay me to run Spark myself these days (and I used to work for the biggest Hadoop vendor in the mid 2010s doing a lot of Spark!)

Re: Data engineering and software engineering are converging

#15

Post by RobinL »

zamalek wrote: post #11
RobinL wrote: post #8

Earlier quoted context omitted.

I think this may be a databricks thing? From what I've seen there's a gap between data engineers forced to use databricks and everyone else. From what I've seen, at least how it's used in practice, databricks seems to result in a mess of notebooks with poor dependency and version management.

Interesting, databricks has been my first exposure to DE at scale and it does seem to solve many problems (even though it sounds like it's causing some). So what does everyone else do? Run spark etc. themselves?

We use aws glue for spark (but are increasingly moving towards duckdb because it's faster for our workloads and easier to test and deploy).

For Spark, glue works quite well. We use it as 'spark as a service', keeping our code as close to vanilla pyspark as possible. This leaves us free to write our code in normal python files, write our own (tested) libraries which are used in our jobs, use GitHub for version control and ci and so on

Re: Data engineering and software engineering are converging

#16

Post by isaacremuant »

CalRobert wrote: post #5

Data engineering was software engineering from the very beginning. Then a bunch of business analysts who didn't know anything about writing software got jealous and said that if you knew SQL/DBT you were a data engineer. I've had to explain too many times that yes, indeed, I can set up a CI/CD pipeline or set up kafka or deploy Dagster on ECS, to the point where I think I need to change my title just to not be cheape…

Agreed. Weird distinction to pay less to people who did certain things and you could a high variance between "data engineers". Some who had only done a course and others that had extensive knowledge of software engineering practices were considered the same.

Ridiculous.

Re: Data engineering and software engineering are converging

#17

Post by flexiflex »

giantg2 wrote: post #3

I've never really seen the distinction between data and software engineering. It's more like front-end vs backend. If you're a data engineer and it's all no code tooling, then you're just an analyst or something.

When I worked at bigCo , it was a totally different world. Data engineers used data platform tools to do data work, usually for data’s sake. Software teams trying to build stuff with data had to finagle their way onto roadmaps.

Re: Data engineering and software engineering are converging

#18

Post by craneca0 »

CalRobert wrote: post #5

Data engineering was software engineering from the very beginning. Then a bunch of business analysts who didn't know anything about writing software got jealous and said that if you knew SQL/DBT you were a data engineer. I've had to explain too many times that yes, indeed, I can set up a CI/CD pipeline or set up kafka or deploy Dagster on ECS, to the point where I think I need to change my title just to not be cheape…

yeah, i've seen large fortune 100 data and analytics orgs where the majority of folks with data engineering titles are uncomfortable with even the basics of git.

Re: Data engineering and software engineering are converging

#19

Post by banku_brougham »

If are orchestrating pipelines in airflow or Prefect you are having to write the client software around those engines, and its a lot of python.

Another anecdatum: the data engineers role at Zillow is called "Software Development Engineer, Big Data"

Re: Data engineering and software engineering are converging

#20

Post by craneca0 »

zurfer wrote: post #2

Maybe. On the one side you have something like dbt or Moosestack. On the other hand analytics and data pipelining is still a lot of no code tooling and I doubt it will go away. However I would love to learn more about how other people use coding agents to do DE tasks.

agreed on the presence and stickiness of no-code tooling. but in a future where we want to enable LLMs and agents to do as much of that work as possible, a code-first approach seems far more likely to make that effective. not just because agents are better are writing code than clicking through interfaces (maybe that will change as agents evolve?), but because the SDLC is valuable for agents for the same reasons it's valuable for human developers - collaboration, testing, auditing, versioning, etc.
Post reply on HN