Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Data engineering and software engineering are converging

clickhouse.com

Post reply on HN
Print view View original

41–50 of 68 posts

Re: Data engineering and software engineering are converging

#41

Post by rawgabbit »

hobs wrote: post #40
rawgabbit wrote: post #4

Earlier quoted context omitted.

In Snowflake, I am now writing Python Stored Procedures that make REST API calls to things like Datadog REST API and dumping the JSON into a Snowflake table. I then unpack the JSON and transform it into a normalized table. So far it works reasonably well. This is possible using Snowflake's external access feature. https://docs.snowflake.com/en/developer-guide/external-netwo...

Right, but isn't snowflake like the most expensive way to run python? why not just host the python outside of snowflake and store it there?

It is to take advantage of snowpark.

Re: Data engineering and software engineering are converging

#42

Post by skybrian »

Here’s an argument that college freshmen should be introduced to data science and computer science in the same introductory course. They’ve written a textbook, which seems pretty sensible:

https://cs.brown.edu/~sk/Publications/Papers/Published/kf-da...

Re: Data engineering and software engineering are converging

#43

Post by jochem9 »

zamalek wrote: post #6

One things have seen through my more recent exposure to experienced data engineers is the lack of repeatability rigor (CI/CD, IaC, etc.). There's a lot of doing things in notebooks and calling that production-ready. Databricks has git (GitHub only from what I can tell) integration, but that's just checking out and directly committing to trunk, if it's in git then we have SDLC right, right? It's fucking nuts. Anyone h…

Last time I worked with Databricks you could just create branches in their interface. PRs etc happened in your git provider, which for us was azure devops back then. We also managed some CI/CD.

You're still dealing with notebooks. Back then there was a tool to connect your IDE to a Databricks cluster. That got killed, not sure if they have something new.

Re: Data engineering and software engineering are converging

#44

Post by jochem9 »

One thing that I don't see mentioned but that does bug me: data engineers often use a lot of Python and SQL, even the ones that have heavily adopted software engineering best practices. Yet both languages are not great for this.

Python is dynamically typed, which you can patch a bit with type hints, but it's still easy to go to production with incompatible types, leading to panics in prod. It's uncompiled nature also makes it very slow.

SQL is pretty much impossible to unit test, yet often you will end up with logic that you want to test. E.g. to optimize a query.

For SQL I don't have a solution. It's a 50 year old language that lacks a lot of features you would expect. It's also the defacto standard for database access.

For Python I would say that we should start adopting statically typed compiled languages. Rust has polars as dataframe package, but the language itself isn't that easy to pick up. Go is very easy to learn, but has no serious dataframe package, so you end up doing a lot of that work yourself in goroutines. Maybe there are better options out there.

Re: Data engineering and software engineering are converging

#45

Post by orochimaaru »

jochem9 wrote: post #44

One thing that I don't see mentioned but that does bug me: data engineers often use a lot of Python and SQL, even the ones that have heavily adopted software engineering best practices. Yet both languages are not great for this. Python is dynamically typed, which you can patch a bit with type hints, but it's still easy to go to production with incompatible types, leading to panics in prod. It's uncompiled nature also…

If you’re using some variety of spark for your data engineering then scala is an option too.

In general, choice of language isn’t important - again if you’re using spark your data frame structure schema defines that structure Python or not.

Most folks confuse pandas with “data engineering”. It’s not. Most data engineering is spark.

Re: Data engineering and software engineering are converging

#46

Post by greekorich »

jochem9 wrote: post #44

One thing that I don't see mentioned but that does bug me: data engineers often use a lot of Python and SQL, even the ones that have heavily adopted software engineering best practices. Yet both languages are not great for this. Python is dynamically typed, which you can patch a bit with type hints, but it's still easy to go to production with incompatible types, leading to panics in prod. It's uncompiled nature also…

I've been a professional java dev for a decade. I've written a little python, clojure, lots of JS/TS/Node.

SQL is the most beautiful, expressive, get stuff done language I've used.

It is perfect for whatever data engineering is defined as.

Re: Data engineering and software engineering are converging

#47

Post by rovr138 »

orochimaaru wrote: post #45
jochem9 wrote: post #44

One thing that I don't see mentioned but that does bug me: data engineers often use a lot of Python and SQL, even the ones that have heavily adopted software engineering best practices. Yet both languages are not great for this. Python is dynamically typed, which you can patch a bit with type hints, but it's still easy to go to production with incompatible types, leading to panics in prod. It's uncompiled nature also…

If you’re using some variety of spark for your data engineering then scala is an option too. In general, choice of language isn’t important - again if you’re using spark your data frame structure schema defines that structure Python or not. Most folks confuse pandas with “data engineering”. It’s not. Most data engineering is spark.

in spark, doesn't pyspark and sql both still get translated to scala?

Re: Data engineering and software engineering are converging

#48

Post by Foobar8568 »

craneca0 wrote: post #18
CalRobert wrote: post #5

Data engineering was software engineering from the very beginning. Then a bunch of business analysts who didn't know anything about writing software got jealous and said that if you knew SQL/DBT you were a data engineer. I've had to explain too many times that yes, indeed, I can set up a CI/CD pipeline or set up kafka or deploy Dagster on ECS, to the point where I think I need to change my title just to not be cheape…

yeah, i've seen large fortune 100 data and analytics orgs where the majority of folks with data engineering titles are uncomfortable with even the basics of git.

Or the basics of SQL...

Re: Data engineering and software engineering are converging

#49

Post by sbrother »

jochem9 wrote: post #44

One thing that I don't see mentioned but that does bug me: data engineers often use a lot of Python and SQL, even the ones that have heavily adopted software engineering best practices. Yet both languages are not great for this. Python is dynamically typed, which you can patch a bit with type hints, but it's still easy to go to production with incompatible types, leading to panics in prod. It's uncompiled nature also…

When I was most recently at Google (2021-ish) my team owned a bunch of SQL Pipelines that had fairly effective SQL tests. Not my favorite thing to work on, but it was a productive way to transform data. There are lots of open source versions of the same idea, but I have yet to see them accompanied with ergonomic testing. Any recommendations or pointers to open source SQL testing frameworks?

Re: Data engineering and software engineering are converging

#50

Post by antupis »

greekorich wrote: post #46
jochem9 wrote: post #44

One thing that I don't see mentioned but that does bug me: data engineers often use a lot of Python and SQL, even the ones that have heavily adopted software engineering best practices. Yet both languages are not great for this. Python is dynamically typed, which you can patch a bit with type hints, but it's still easy to go to production with incompatible types, leading to panics in prod. It's uncompiled nature also…

I've been a professional java dev for a decade. I've written a little python, clojure, lots of JS/TS/Node. SQL is the most beautiful, expressive, get stuff done language I've used. It is perfect for whatever data engineering is defined as.

SQL is beautiful when it works but when it doesn’t you end up with some abomination eg if you need some kind dynamic query.
Post reply on HN