What Is Dbt and Why Are Companies Using It? • Hacker News Board

Re: What Is Dbt and Why Are Companies Using It?

#71

Post by glogla » Fri, Dec 03, 2021, 5:37 PM UTC

tomnipotent wrote: post #66

xyzzy_plugh wrote: post #6

I can't argue that dbt isn't great -- it is. It is, however , unfortunate that Python is still the dominate lingua franca here. Between all the Python data tools and JSON being shuffled around ala singer.io, I just can't help but think there is a huge opportunity awaiting the engineers who build a company around an open source set of products using rigorous languages and schemas. It's a goddamn madhouse.

> using rigorous languages and schemas And what value does it add? A vast majority of companies are working with Changing singer.io to require Thrift or protobuf schemas isn't going to add the value you think it is. How data is shuffled between systems is considerably less important and time consuming than figuring out how to put that data to work.

Singer and dbt are different - dbt orchestrates stuff (evaluates macros and runs SQLs) while Singer actually has the data flow through it. So rewriting Singer in something fast (I guess Java since even the most obscure database has JDBC driver for it) would definitely help.

It would only help once you start shuttling around terabytes.

Re: What Is Dbt and Why Are Companies Using It?

#72

Post by tomnipotent » Fri, Dec 03, 2021, 10:29 PM UTC

glogla wrote: post #71

tomnipotent wrote: post #66

Earlier quoted context omitted.

> using rigorous languages and schemas And what value does it add? A vast majority of companies are working with Changing singer.io to require Thrift or protobuf schemas isn't going to add the value you think it is. How data is shuffled between systems is considerably less important and time consuming than figuring out how to put that data to work.

Singer and dbt are different - dbt orchestrates stuff (evaluates macros and runs SQLs) while Singer actually has the data flow through it. So rewriting Singer in something fast (I guess Java since even the most obscure database has JDBC driver for it) would definitely help. It would only help once you start shuttling around terabytes.

These tools were not built for multi-tb workloads, and pointing out that as a deficiency when it was clearly not a design goal is a misleading argument.

Re: What Is Dbt and Why Are Companies Using It?

#73

Post by glogla » Fri, Dec 03, 2021, 11:10 PM UTC

Is there dbt competition? I know of https://dataform.co that looked pretty similar, but that got bought by Google so I don't know if it would be a good investment now.

Any other?

Re: What Is Dbt and Why Are Companies Using It?

#74

Post by norton120 » Tue, Dec 07, 2021, 7:34 PM UTC

One of the better ways I've seen dbt described is very similar to an MVC framework for data (I didn't see that mentioned yet, if it has been sorry I missed it). Where a rough mapping would be:

model >> source

controller >> ephemeral model

view >> materialized (view, table, incremental) model

Like many MVCs it provides abstractions and helpers for common tasks such as performing incremental updates, snapshotting time series, common manipulations like creating date spines. Like many MVCs it supports a robust ecosystem of plugins that make it easy to re-use stable transforms for common datasets. Where Django passes you the request object instead of hand-parsing http responses, dbt allows you to loop through a set of derived schemas and dry out your SQL code.

You would generally use dbt as a transform framework for the same reason you'd use Ruby on Rails or Django etc as a web framework - because it provides you with a ton of otherwise repetitive non-differentiating code pre-baked and ready to go. You could keep a folder of sql files you arbitrarily run, and you could roll your own web framework from scratch. Personally I wouldn't do those things.