Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What Is Dbt and Why Are Companies Using It?

seattledataguy.substack.com

Post reply on HN
Print view View original

51–60 of 74 posts

Re: What Is Dbt and Why Are Companies Using It?

#51

Post by adeelk93 »

bxji wrote: post #46

I’m still a little confused on what dbt does after reading the article. Is it like Trino that generates materialized views as output, with built-in version control and other features?

I'd list the DAG and testing as core features as well, but yeah, basically. In a very transparent and non-magical way (I despise magical data solutions). If that summary doesn't resonate with you, it's probably not the tool for you. No need to force it.

I'm not familiar with Trino at all, but that sounds like a specific database. dbt is not a database, it is tooling for working with databases.

Re: What Is Dbt and Why Are Companies Using It?

#52

Post by swuecho »

elchief wrote: post #47
thejosh wrote: post #38

Earlier quoted context omitted.

nope, it has lots of other fantastic features such as a dag, macros, etc.

make is the OG DAG. And has macros

In theory, you could write dbt in make. Actually, Whenever I introduce to others about workflow concepts (DAG), I tell them it is similar to how make works.

Re: What Is Dbt and Why Are Companies Using It?

#53

Post by thejosh »

theplague42 wrote: post #42
thejosh wrote: post #37

Earlier quoted context omitted.

Exactly what we've been doing for 2+ years at my job. It's amazing. Snowflake is amazing, but watch out for search optimization costs (it's great for append only), left joins taking FOREVER (avoid left joins as much as possible for large datasets).

What makes left joins perform poorly in Snowflake?

It has to join each part of the previous join to the next join, and if you have a lot of joins this can get out of hand.

We have a lot of joins in our final fct orders from our intermediate table, and looks like this:

from foo left join bar on bar.common_id = foo.common_id left join baz on baz.common_id = foo.common_id left join qux on qux.common_id = foo.common_id left join waldo on waldo.common_id = foo.common_id

So waldo joins to qux, which joins to qux... I call it a "staircase join", as that's what it looks like in the SF profiler.

Re: What Is Dbt and Why Are Companies Using It?

#54

Post by andygrd »

Having used it for a little over a year now, I can say that it's strengths may lie in getting junior developer's code more free of bugs and dependency issues. It's just additional overhead when you're trying to do anything more complex, like a Kimball type II slowly changing dimension - then it's just a blocker. Unfortunately, as it becomes a defacto build environment, it's limitations start getting applied to everyone.

Re: What Is Dbt and Why Are Companies Using It?

#55

Post by benjaminwootton »

dang wrote: post #15
elchief wrote: post #14

So, it's like a makefile with some SQL to do ELT?

" Please don't post shallow dismissals, especially of other people's work. A good critical comment teaches us something. " https://news.ycombinator.com/newsguidelines.html Edit: if you didn't mean it dismissively and want to clarify that, I'd be happy to remove this comment. We unfortunately see quite a few comments along the lines of "so basically, this is just super-simple $thing and therefore this is dumb" and I i…

I actually really like DBT and advocated for it. It is however quite a simple tool, just some wrappers and a workflow around SQL transformations. Maybe it’s simplicity is it’s quality, but the OP you replied to isn’t as far off the mark as similar comments.

Re: What Is Dbt and Why Are Companies Using It?

#56

Post by dikei »

bxji wrote: post #46

I’m still a little confused on what dbt does after reading the article. Is it like Trino that generates materialized views as output, with built-in version control and other features?

In your case, dbt would be the tool to manage your Trino view programmatically:

* You define the sql queries that select the input data as models and the dbt scripts (also sql) to combine/manipulate the models.

* On running, dbt will generate the Trino SQL queries to join/transform these models into the final result (also a model).

* Depending on your configuration, any models can be stored as a Trino view or it can be materialized as a data table that's updated every time you re-run the dbt scripts.

Re: What Is Dbt and Why Are Companies Using It?

#57

Post by benjaminwootton »

Not to steal the OPs thunder, but I have also created a few resources on DBT if they help.

A demo of integrating DBT and Snowflake https://timeflow.systems/connecting-dbt-to-snowflake

How DBT DevOps enables data teams: https://timeflow.academy/blog/how-dbt-devops-enables-data-te...

I am also writing a tutorial on DBT, though there are a few rough edges still: https://timeflow.academy/dbt-introduction

Re: What Is Dbt and Why Are Companies Using It?

#58

Post by antruok »

andygrd wrote: post #54

Having used it for a little over a year now, I can say that it's strengths may lie in getting junior developer's code more free of bugs and dependency issues. It's just additional overhead when you're trying to do anything more complex, like a Kimball type II slowly changing dimension - then it's just a blocker. Unfortunately, as it becomes a defacto build environment, it's limitations start getting applied to everyo…

The snapshots feature should handle the slowly changing dimensions. But are you saying it's not flexible enough for certain edge cases?

Re: What Is Dbt and Why Are Companies Using It?

#59

Post by andygrd »

antruok wrote: post #58
andygrd wrote: post #54

Having used it for a little over a year now, I can say that it's strengths may lie in getting junior developer's code more free of bugs and dependency issues. It's just additional overhead when you're trying to do anything more complex, like a Kimball type II slowly changing dimension - then it's just a blocker. Unfortunately, as it becomes a defacto build environment, it's limitations start getting applied to everyo…

The snapshots feature should handle the slowly changing dimensions. But are you saying it's not flexible enough for certain edge cases?

Yeah, dbt snapshots do a row hash and update if anything in the row changes. We had a source table that had a bunch of daily changing timestamps, e.g. "load date", that we needed to ignore, and focus on a business key. Dbt was an utter torment to try get this going. Ended up building a more elegant framework without it.
Post reply on HN