Live data from Hacker News

Dbt – Incremental but Incomplete

tobikodata.com

11–20 of 20 posts

Re: Dbt – Incremental but Incomplete

#11
post #10
post #6

Can someone who understands it explain what dbt is and how it is used. I hear a lot about it but I just haven't figured out what it is useful for.

Basically people are constantly calculating metrics based on existing tables. Think something as simple as a moving average or the sum of two separate columns in a table. Once upon a time you would set up a cronjob and populate these every day as a SQL query in some python or Perl script. Dbt introduced a language for managing these “metrics” at scale including the ability to use variables and more complex templates…

[deleted]

Re: Dbt – Incremental but Incomplete

#12

Is anyone using SQLMesh in production? I love “lessons learned” tools which have the opportunity to improve core design after seeing the weak points of the initial product in the space. That being said, I hate being an early adopter, so will let others determine if the new tool has an entirely novel set of shortcomings vs dbt.

There are many teams using SQLMesh in production. Fivetran, Harness, Hopper, Pitchbook to name a few. You can read some case studies here https://tobikodata.com/harness.html or join Slack to meet with folks to learn more about their experiences.

How does Fivetran use SQLMesh?

Re: Dbt – Incremental but Incomplete

#13
post #7
post #6

Can someone who understands it explain what dbt is and how it is used. I hear a lot about it but I just haven't figured out what it is useful for.

I am not sure if it is that popular these days. Couple of years ago it was pretty popular.

sounds like a typical hype-tech lifecycle.

Re: Dbt – Incremental but Incomplete

#14
post #7
post #6

Can someone who understands it explain what dbt is and how it is used. I hear a lot about it but I just haven't figured out what it is useful for.

I am not sure if it is that popular these days. Couple of years ago it was pretty popular.

The hype may have gone down, but it's usage is good. It's used where I work. It has a slack channel that's pretty busy.

Re: Dbt – Incremental but Incomplete

#15
post #12

Earlier quoted context omitted.

There are many teams using SQLMesh in production. Fivetran, Harness, Hopper, Pitchbook to name a few. You can read some case studies here https://tobikodata.com/harness.html or join Slack to meet with folks to learn more about their experiences.

How does Fivetran use SQLMesh?

They're using it for data transformation.They're long time dbt users, but are switching to SQLMesh because it's extremely efficient, provides a better development experience, and can help them become warehouse agnostic.

Re: Dbt – Incremental but Incomplete

#16

I really wish data engineers didn't have to hand-roll incremental materialization in 2024. This is really hard stuff to get right (as the post outlines) but it is absolutely critical to keeping latency and costs down if you're going to go all in on deep, layered, fine-grained transformations (which still seems to me to be the best way to scale a large / complex analytics stack). My prediction a few years back was tha…

I wholeheartedly agree. When I worked at Shopify, we had to hand-roll our incremental data models using Spark, and the complexity of managing deep DAGs made tasks like backfilling and refactoring a huge pain. Tools like dbt and SQLMesh face similar challenges.

The chaos of existing approaches was a large part of what drove me to join Materialize. With Materialize, you can use dbt on “easy-mode”, while Materialize handles incremental logic, removing the usual headaches around processing time and keeping everything up to date within a second or two.

I recently gave a talk at Data Council about this unlock, it’s total magic: https://youtu.be/pLb5sFZ7nWw

For anyone interested, my colleague Seth also discussed this in a recent blog post: https://materialize.com/blog/migrating-postgres-materialize/

Re: Dbt – Incremental but Incomplete

#17
Side question: For ETL development based on PostGIS and pgRouting, which tool is more suitable? In other words, which one is easier to work with for geometric data? (e.g., visual test case data display, etc.)

Re: Dbt – Incremental but Incomplete

#18
post #6

Can someone who understands it explain what dbt is and how it is used. I hear a lot about it but I just haven't figured out what it is useful for.

I built the first half of my career as "a guy who knows SQL" (and Excel macros but I digress). I then rode the early wave of Analytics Engineering.

dbt is kinda like Vite (dbt = data build tool) for folks working with data warehouses. Their biggest contribution was a mindset shift that applied principles of the SDLC to the traditional BI/Analytics space.

Almost overnight, analysts went from building business logic in GUIs like Talend or Tableau to code-based models (SQL) checked into git repos instead. It took what Looker was doing with LookML and generalized it across the BI stack.

This shift (+ associated tooling) resulted in less brittle data pipelines, increased uptime for dashboards/reporting, and more sanity when working with more than 2-3 people in a data environment.

Imagine a situation where you're at an e-commerce company and need to reconcile orders from Woocommerce with shipments in ShipStation, returns from tickets in HubSpot, and refunds issued in Stripe. dbt simplifies the management of the relationships between these various systems.

Based on this, you can build data models that allow you and, increasingly, your business stakeholders to answer questions like "Which SKUs have seen an uptick in refunds due to reason X this quarter?" and "Where were they shipped?"

The benefit of having standard abstractions means you can build metrics on top of the models as [gkapur](https://news.ycombinator.com/item?id=41853925) mentions such that "revenue" is the same when marketing pulls it for calculating CAC as when finance pulls it their monthly reports, etc.

Re: Dbt – Incremental but Incomplete

#19
post #7

Earlier quoted context omitted.

I am not sure if it is that popular these days. Couple of years ago it was pretty popular.

The hype may have gone down, but it's usage is good. It's used where I work. It has a slack channel that's pretty busy.

dbt isn't going anywhere. It's the standard.

That said, SQLMesh and other tools are pretty interesting and I look forward to new growth in the space.

Re: Dbt – Incremental but Incomplete

#20

I really wish data engineers didn't have to hand-roll incremental materialization in 2024. This is really hard stuff to get right (as the post outlines) but it is absolutely critical to keeping latency and costs down if you're going to go all in on deep, layered, fine-grained transformations (which still seems to me to be the best way to scale a large / complex analytics stack). My prediction a few years back was tha…

The Databricks AutoLoader for Delta Live Tables with Checkpointing and Watermarking comes to your rescue.
Post reply on HN