11–20 of 37 posts
This is a joke. You don’t “level up beyond ETL” by speeding up SQL queries. In fact, there’s pretty much nothing about ETL in that article.
Perhaps start by explaining what ETL stands for and what it is.
Acronyms that aren't defined only hurt.
This is a joke. You don’t “level up beyond ETL” by speeding up SQL queries. In fact, there’s pretty much nothing about ETL in that article.
Thought this too. I liked article, had some interesting points. But never found the ETL part. Kept wondering when I'd scroll and hit the ETL section.
Perhaps start by explaining what ETL stands for and what it is.
It sounds to me like the author got thrown to the wolves in an environment of what data engineering looked like before "big data" and ML took off (and before it was even really called data engineering). There are a lot of enterprises that are still working in this mode because they are not Google and they don't have the same level of sophistication and automation when it comes to this stuff.
There is some bad information no doubt in the article, but if we're being charitable, it feels like it's someone who took a wrong turn somewhere and is struggling to find their feet in an unfamiliar place without the proper guidance and mentorship and that's a bit admirable at least that they're trying on their own.
There is no direct bearing on ETL in the article, aside from the focus on SQL queries and data validation hints that they might be talking about ELT (Extract-Load-Transform) as the level beyond ETL, but it's not clearly explained. It's clear to me that they are at the start of their journey and they are gonna learn things the hard way without guidance from someone more experienced.
"The tricky part is when context goes ‘over the wall’ - the consumer finds an issue with the data but the producer is not familiar with the domain to see it’s a problem. Over time, information transfer between the two sides becomes the bottleneck and issues pile up."
Data engineering teams tend to have the least context and the most responsibility to fix everything.
... proceeds to rewrite the stack in Python.
Personally, I've never seen SQL unit tests/asserts demonstrate value in practice. The vast majority of tests are simple uniqueness or non-null tests that would only catch the sloppiest mistakes. I feel like it's emblematic of a Data Engineering culture that is more interested in the Engineering part than the actual Data products they are supposed to be building.
Do you mind expanding why? As someone currently looking for ways to integrate standard software engineering practices into (analytical) SQL code, I am curious about your reasoning.
> The vast majority of tests are simple uniqueness or non-null tests that would only catch the sloppiest mistakes
But don't they still happen? Perhaps they're not useful now, but may reveal clear problems when the original implementation is stretched in ways that weren't anticipated -- which is a common argument in favor of testing.
To add to this, I've been keeping an eye on sqlmesh and I like the distinction they make between testing [1] (asserting logic) and auditing [2] (validating expectations on the data). Their builtin audits go beyond "not-null" and "unique", including statistical checks, for example.
[1] https://sqlmesh.readthedocs.io/en/stable/concepts/tests/
[2] https://sqlmesh.readthedocs.io/en/stable/concepts/audits/