dbt is the quickest way to watch your cloud costs skyrocket. I really do hope people don’t think this is a way to not have dedicated data engineers on staff and instead “empower” analysts to do things for themselves.
Could you expand on this point?
What Is Dbt and Why Are Companies Using It?
61–70 of 74 posts
Re: What Is Dbt and Why Are Companies Using It?
#62I have been trying to migrate our legacy ETLs to DBT, which is mostly strings peppered around in python code.
I like the macros, templates, the visualisation tool ..... and would like to make our code as DBT compatible as possible.
However, the DB we currently use (Vertica) does not have official support. There are dependency problems installing all of DBT. All I would like to get for my first milestone is to use refs and macros and better organise the SQLs. It is good enough if it generates standard SQL, that I can run outside DBT.
My wish:
I wish I could install select DBT packages just for what I need (templates, macros, refs, dependency) and still make sure I can gradually achieve DBT compatibility. At the moment it looks like all or nothing.
Re: What Is Dbt and Why Are Companies Using It?
#63[deleted]
Re: What Is Dbt and Why Are Companies Using It?
#64Earlier quoted context omitted.
The snapshots feature should handle the slowly changing dimensions. But are you saying it's not flexible enough for certain edge cases?
Yeah, dbt snapshots do a row hash and update if anything in the row changes. We had a source table that had a bunch of daily changing timestamps, e.g. "load date", that we needed to ignore, and focus on a business key. Dbt was an utter torment to try get this going. Ended up building a more elegant framework without it.
So apparently the check_cols parameter didn't help either?
I'm trying to learn about the critical pain points of dbt, and this case seems interesting.
Re: What Is Dbt and Why Are Companies Using It?
#65I'm the CTO of a 15 person startup and dbt (along with Mode, Stitch and BigQuery) has been a superpower allowing me to moonlight as a CDO (or some shitty pastiche of one) and push forward our data usage and competence in ways I just didn't think possible. Modern data stack is a marvel in what it unlocks.
Re: What Is Dbt and Why Are Companies Using It?
#66I can't argue that dbt isn't great -- it is. It is, however , unfortunate that Python is still the dominate lingua franca here. Between all the Python data tools and JSON being shuffled around ala singer.io, I just can't help but think there is a huge opportunity awaiting the engineers who build a company around an open source set of products using rigorous languages and schemas. It's a goddamn madhouse.
> using rigorous languages and schemas
And what value does it add?
A vast majority of companies are working with Changing singer.io to require Thrift or protobuf schemas isn't going to add the value you think it is. How data is shuffled between systems is considerably less important and time consuming than figuring out how to put that data to work.
Re: What Is Dbt and Why Are Companies Using It?
#67I have been trying to migrate our legacy ETLs to DBT, which is mostly strings peppered around in python code. I like the macros, templates, the visualisation tool ..... and would like to make our code as DBT compatible as possible. However, the DB we currently use (Vertica) does not have official support. There are dependency problems installing all of DBT. All I would like to get for my first milestone is to use ref…
If your transformations are already in Python scripts wouldn’t you be better off with Airflow or Luigi? dbt seems to be aimed at SQL users primarily.
Re: What Is Dbt and Why Are Companies Using It?
#68dbt is the quickest way to watch your cloud costs skyrocket. I really do hope people don’t think this is a way to not have dedicated data engineers on staff and instead “empower” analysts to do things for themselves.
[dbt Labs employee here]
Cloud costs increasing could be a sign of more data being utilized productively. In any way, a mixture of data engineers and analysts is a healthy way to scale a dbt project to increase speed-of-delivery of analytics requests vs cloud costs. We at dbt Labs encourage the "analytics engineering" mindset to bring software engineering best practices into the dbt analytics workflow (git version control, code reviews), and so cloud cost considerations should be incorporated into mature dbt development practices.
Re: What Is Dbt and Why Are Companies Using It?
#69I have been trying to migrate our legacy ETLs to DBT, which is mostly strings peppered around in python code. I like the macros, templates, the visualisation tool ..... and would like to make our code as DBT compatible as possible. However, the DB we currently use (Vertica) does not have official support. There are dependency problems installing all of DBT. All I would like to get for my first milestone is to use ref…
If your transformations are already in Python scripts wouldn’t you be better off with Airflow or Luigi? dbt seems to be aimed at SQL users primarily.
They are essentially SQL transformation at the reporting stage. Python is just the glue code, a lot of attention has gone into the connectors but no so much into the complexity within the SQL.
The problem I would like to address is complex SQL written as strings. Some parts of these repeat over multiple reports, column transformations, look ups, joins with dimensions .....
Re: What Is Dbt and Why Are Companies Using It?
#70Not to steal the OPs thunder, but I have also created a few resources on DBT if they help. A demo of integrating DBT and Snowflake https://timeflow.systems/connecting-dbt-to-snowflake How DBT DevOps enables data teams: https://timeflow.academy/blog/how-dbt-devops-enables-data-te... I am also writing a tutorial on DBT, though there are a few rough edges still: https://timeflow.academy/dbt-introduction
>A demo of integrating DBT and Snowflake https://timeflow.systems/connecting-dbt-to-snowflake
This link returns a 404