Live data from Hacker News

How to level up beyond ETLs

ezzeriesa.com

21–30 of 37 posts

Re: How to level up beyond ETLs

#21
post #14

Personally, I've never seen SQL unit tests/asserts demonstrate value in practice. The vast majority of tests are simple uniqueness or non-null tests that would only catch the sloppiest mistakes. I feel like it's emblematic of a Data Engineering culture that is more interested in the Engineering part than the actual Data products they are supposed to be building.

We got a lot of value using SQL unit tests on very complex queries with lots of edge cases. There was one which involved breaking up events into session (a session is a single uninterrupted 'seating' using the product, an interruption being 30 mins gap or more). Re: edge cases, we'll check sessions are broken up as expected when the user changes, when the browser changes, when it goes over midnight UTC etc.

The table was ~250mm rows per day. The alternative would be doing it in memory; Python unit tests do feel a bit more natural but for a table that size we decided it was best to do it on disk / let the DBMS deal with it. Perhaps Spark is an idea but then there's the trade-off of customers losing context.

It also helped us 'refactor with confidence' - we can happily change the query to incorporate new use cases while knowing the core logic is still sound.

Re: How to level up beyond ETLs

#22
post #6

Isn't this very very basic? I'm not seeing the "beyond" element, it's more like "How to level up to almost not zero".

Fair point. Perhaps title can be 'How to keep things interesting when you're not happy serving customer requests all day'. On the flip side, not suggesting that you don't serve customer requests either so more of a way to keep your head up while doing day-to-day.

Re: How to level up beyond ETLs

#23

Writing better code does not solve the listed problem of low context. "The tricky part is when context goes ‘over the wall’ - the consumer finds an issue with the data but the producer is not familiar with the domain to see it’s a problem. Over time, information transfer between the two sides becomes the bottleneck and issues pile up." Data engineering teams tend to have the least context and the most responsibility…

It can often feel like a thankless job :cry:

Re: How to level up beyond ETLs

#24
post #10

This is a joke. You don’t “level up beyond ETL” by speeding up SQL queries. In fact, there’s pretty much nothing about ETL in that article.

Thought this too. I liked article, had some interesting points. But never found the ETL part. Kept wondering when I'd scroll and hit the ETL section.

Thanks! There's ETL in the traditional sense of moving production data to the data warehouse (hence the 'reverse ETL' moving data back into prod), as well as the term increasingly being used for data pipelines in general.

On the former, we had custom tooling at the time of writing but now use GCP Dataflow (which I can say I'm rather partial to).

Re: How to level up beyond ETLs

#25
post #3

What an honest an refreshing perspective! I loved the quote from Donald Knuth: If you find that you’re spending almost all your time on theory, start turning some attention to practical things; it will improve your theories. If you find that you’re spending almost all your time on practice, start turning some attention to theoretical things; it will improve your practice.

Totally. That being said I'm on 'soft skills' phase now :joy:

Re: How to level up beyond ETLs

#26
post #17

Perhaps start by explaining what ETL stands for and what it is.

It seems like the article is written by someone just starting to get into the data engineering subfield and they thought they were going to be writing python (pyspark is my guess) to support some kind of ML effort, but they got saddled with a bunch of SQL/data warehousing stuff to support business intelligence/analytics instead. I'd say normally what you say makes sense especially when you're pulling in abbreviations…

> There is some bad information no doubt in the article

Could you share more specific details? Happy to look over / revise where needed.

More broadly is the issue of the gap of what you think the role is, and what the role actually is when you join. There are definitely cases where this is accidental. The best way I can think of to close the gap is to maybe do a short-term contract, but may be challenging to do under time constraints etc.

Re: How to level up beyond ETLs

#27
post #14

Personally, I've never seen SQL unit tests/asserts demonstrate value in practice. The vast majority of tests are simple uniqueness or non-null tests that would only catch the sloppiest mistakes. I feel like it's emblematic of a Data Engineering culture that is more interested in the Engineering part than the actual Data products they are supposed to be building.

> Personally, I've never seen SQL unit tests/asserts demonstrate value in practice. Do you mind expanding why? As someone currently looking for ways to integrate standard software engineering practices into (analytical) SQL code, I am curious about your reasoning. > The vast majority of tests are simple uniqueness or non-null tests that would only catch the sloppiest mistakes But don't they still happen? Perhaps they…

Totally. Wouldn't recommend all queries have it, but there are some critical tables that should.

Re: How to level up beyond ETLs

#28

Not sure if it’s just me, but the whole mention of the Stitch Fix blog post and ‘democratising data pipelines’ really threw me off. > The idea here is the engineering team creates ‘Lego blocks’ that consumers then assemble into end-to-end data pipelines. Might be my personal biases, but it doesn’t really seem to set the scene for the rest of content? The rest of it was good enough common sense “non exhaustive list of…

The best practices book definitely changes quickly in a new area, I can imagine a new / different set on the back of data pipelines for generative models.

Re: How to level up beyond ETLs

#29
post #14

Personally, I've never seen SQL unit tests/asserts demonstrate value in practice. The vast majority of tests are simple uniqueness or non-null tests that would only catch the sloppiest mistakes. I feel like it's emblematic of a Data Engineering culture that is more interested in the Engineering part than the actual Data products they are supposed to be building.

> Personally, I've never seen SQL unit tests/asserts demonstrate value in practice. Do you mind expanding why? As someone currently looking for ways to integrate standard software engineering practices into (analytical) SQL code, I am curious about your reasoning. > The vast majority of tests are simple uniqueness or non-null tests that would only catch the sloppiest mistakes But don't they still happen? Perhaps they…

Sure thing. Thinking back, the "data issues" I've encountered roughly fall into two buckets - something changed in the source data, and something broke in the code.

My first reservation with tests is that the vast majority of issues fall into the first bucket. Audits are a solution to the source data change problems (though trying to anticipate the myriad ways in which source data can change feels almost impossible, such that audits end up becoming more of a "lets not fuck it up THAT way again"), but Tests are not. Yet writing and debugging tests takes as much or more effort.

My other reservation is that generally, I see a lot of VERY basic tests that only check for a limited, very simple subset of possible errors. So even for data issues that fall into the latter bucket, the tests aren't catching any issues. Things like adding something to a where clause that references a column from a LEFT JOINed table or using "= TRUE" instead of "IS TRUE" generally isn't something that test cases or asserts will catch (in my experience).

I understand that ensuring code quality requires redundant layers of checks, so tests should act as an additional safety net for developer skill and code reviews. But I also think that teams have limited bandwidth, and I would prefer efforts be focused on higher ROI activities than writing tests.*

*Which is not to say I write SQL without testing it. I prefer to think about each change I am make as requiring a unique set of checks, based on what could go wrong with the specific changes I'm making, and manually test outputs accordingly. ie run with old and new versions of the code, and comparing to see if row counts change (or don't change) as expected.

Re: How to level up beyond ETLs

#30
post #14

Personally, I've never seen SQL unit tests/asserts demonstrate value in practice. The vast majority of tests are simple uniqueness or non-null tests that would only catch the sloppiest mistakes. I feel like it's emblematic of a Data Engineering culture that is more interested in the Engineering part than the actual Data products they are supposed to be building.

> Personally, I've never seen SQL unit tests/asserts demonstrate value in practice. Do you mind expanding why? As someone currently looking for ways to integrate standard software engineering practices into (analytical) SQL code, I am curious about your reasoning. > The vast majority of tests are simple uniqueness or non-null tests that would only catch the sloppiest mistakes But don't they still happen? Perhaps they…

> clear problems when the original implementation is stretched in ways that weren't anticipated -- which is a common argument in favor of testing.

On this point in particular, I'm not sure of this is happens very often in data engineering (especially with data transforms), since I don't think tables experience the type of scope creep the way app components or APIs might. Once a table is shipped, almost all every subsequent change is either adding/removing columns (which I think should be written in a way that means it's impossible to change the grain), fixing bugs (in which case tests are not relevant), or internally refactoring for performance (tests can help, but usually only cover very basic issues).

The latter case is actually one where I think automated, generic testing would be helpful, but I'm not aware of any existing tools make easy? Ideally, I would want a test suite to run new and old versions of the code in parallel, and confirm that outputs are unchanged.

Post reply on HN