Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What Is Dbt and Why Are Companies Using It?

seattledataguy.substack.com

Post reply on HN
Print view View original

21–30 of 74 posts

Re: What Is Dbt and Why Are Companies Using It?

#21

Post by jgraettinger1 »

xyzzy_plugh wrote: post #6

I can't argue that dbt isn't great -- it is. It is, however , unfortunate that Python is still the dominate lingua franca here. Between all the Python data tools and JSON being shuffled around ala singer.io, I just can't help but think there is a huge opportunity awaiting the engineers who build a company around an open source set of products using rigorous languages and schemas. It's a goddamn madhouse.

Our own approach is to keep singer.io et al and JSON (flexible! streaming capable!), but deeply invest in JSON Schema to a) statically infer and map into other kinds of schema on your behalf -- TypeScript, Elastic Search, and SQL DDL so far -- and b) optimize validation so it's fast enough to be in the high-scale critical path. If you validate every document before it's read or written by your hacky transform, you've gone a long way towards limiting the bug blast radius.

Re: What Is Dbt and Why Are Companies Using It?

#22

Post by swuecho »

22c wrote: post #18
dang wrote: post #15

Earlier quoted context omitted.

" Please don't post shallow dismissals, especially of other people's work. A good critical comment teaches us something. " https://news.ycombinator.com/newsguidelines.html Edit: if you didn't mean it dismissively and want to clarify that, I'd be happy to remove this comment. We unfortunately see quite a few comments along the lines of "so basically, this is just super-simple $thing and therefore this is dumb" and I i…

Is it a dismissal or is it someone asking for affirmation on their understanding of the product? I think it can be read both ways.

If it is for for affirmation. The answer is Yes. I even write a similar tool using Scons. However, dbt is far better.

Re: What Is Dbt and Why Are Companies Using It?

#23

Post by 41b696ef1113 »

orzig wrote: post #5

I am a user, and what sets them apart even beyond the product is the amazing community they have built that celebrates the unsexy aspects of data science. Just watch some videos from their annual conference, people are so excited to talk about rationalizing business logic

Evidently, they have a free conference next week[0].

[0] https://coalesce.getdbt.com

Re: What Is Dbt and Why Are Companies Using It?

#24

Post by lbotos »

xyzzy_plugh wrote: post #6

I can't argue that dbt isn't great -- it is. It is, however , unfortunate that Python is still the dominate lingua franca here. Between all the Python data tools and JSON being shuffled around ala singer.io, I just can't help but think there is a huge opportunity awaiting the engineers who build a company around an open source set of products using rigorous languages and schemas. It's a goddamn madhouse.

I think this is what meltano is trying to help with?

https://www.meltano.com/

Re: What Is Dbt and Why Are Companies Using It?

#27

Post by killingtime74 »

xyzzy_plugh wrote: post #6

I can't argue that dbt isn't great -- it is. It is, however , unfortunate that Python is still the dominate lingua franca here. Between all the Python data tools and JSON being shuffled around ala singer.io, I just can't help but think there is a huge opportunity awaiting the engineers who build a company around an open source set of products using rigorous languages and schemas. It's a goddamn madhouse.

There are only languages people hate and ones no one use

Re: What Is Dbt and Why Are Companies Using It?

#28

Post by beckingz »

Data Build Tool makes it easier to get data analysts involved, which helps bring more analytics code into version control. This is a really really good thing given that most data analysts are not software engineers and rightfully prioritize putting their analytical skills to use in the most efficient way possible, which tends to result in analysis getting lost in a data swamp.
Post reply on HN