Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Rocky – Rust SQL engine with branches, replay, column lineage

github.com

Post reply on HN
Print view View original

31–40 of 58 posts

Re: Show HN: Rocky – Rust SQL engine with branches, replay, column lineage

#31

Post by thelittlenag »

Xiaoher-C wrote: post #23

The compile-time lineage part is the most interesting bit to me. A lot of “data lineage” tools feel like archaeology after the fact: parse logs, reconstruct what probably happened, then hope it matches reality. Having the compiler know “this column flows into these downstream models” before execution changes the workflow quite a bit. It makes refactors and masking policies much less scary. Do you expose any kind of “…

Same. I worked on an in-house product many years ago now where lineage and provenance were the entire point. Really cool to see this!

Re: Show HN: Rocky – Rust SQL engine with branches, replay, column lineage

#32

Post by hobofan »

PeterWhittaker wrote: post #17

Congrats on the work, but have you considered another name? Naming is hard and always will be: When I first scanned the headline, my initial thought was "that's an interesting area for the Rocky Linux team to explore". After a moment, "wait, no, that's confusing, it's some other Rocky".

I fear that there is an even closer candidate for confusion: RocksDB

Re: Show HN: Rocky – Rust SQL engine with branches, replay, column lineage

#33

Post by hugocorreia90 »

data_ders wrote: post #24

hiya, anders from dbt here. cool project -- I especially love the branching and budgeting options you've built in. both are things that I'd love for the dbt standard to include one day. was it dbt's lack of those feature that inspired you to start this project? It also seems you have an aversion to Jinja, which, believe me, I get! FYI dbt-fusion [1] is going GA next week (though GA for Databricks will come later) Mos…

Hey Anders! Thanks a lot for dropping a comment and show interest in Rocky. Yes! I won't going to lie that Jinja is one of the things that gives me some itches :). But it wasn't the major reason for start building Rocky though.

It all started with the need for auto-generating dbt models from the FiveTran connections I integrate with, then having to hot reload code location in Dagster to discover new assets. All in a zero-touch data pipeline. FiveTran connections are discovered as they're created, assets are materialized as these connections sync.

Auto-generating these dbt models and get the manifest aligned between Dagster code location reloads plus spinning up pods in EKS for each Dagster runs that need to rely on these auto-generated models have some impact on the performance overall, not only in production, but also affects DX in their local environment.

Rocky wasn't born with a "dbt replacement" in mind at all, but it was born to solve a real issue I'm facing. I made sure I can integrate well with dbt as it's in my plans to leverage the awesome work available as dbt packages for FiveTran.

I'll definitely have a look the crates you mentioned! Thank you!

Re: Show HN: Rocky – Rust SQL engine with branches, replay, column lineage

#34

Post by hugocorreia90 »

DoctorOW wrote: post #21
hugocorreia90 wrote: post #8

Earlier quoted context omitted.

Yeah, fair pushback, and yes the intro was AI-assisted. Marketing is not my strength nor I am a native english speaker. I built this in about a month with heavy LLM tooling and the seed comment is part of that. I'm not going to pretend otherwise. The code is what it is. `cargo test --workspace` runs across 19 crates. CI on 5 platforms (macOS ARM/Intel, Linux x86/ARM, Windows). JSON output schemas are codegen-checked…

> I'd rather hear "the code is bad" than "the post sounds AI-written". Of course you would. Reading through and judging the quality of AI output is the largest amount of effort in a world where you can get everything else by prompting. Please internalize this: If you want to be respected you will have to put in effort yourself. There is no way around this.

I truly appreciate your feedback and it's definitely a lesson learned for me. As I said to cmrdporcupine, "The only reason for passing my replies through AI was just because it's my first time posting here and opening a side-project of mine publicly. "

All the engine architecture decisions are mine though and this project came up to solve a real problem in a data pipeline that serves multiple clients, connectors, producers, etc.

Re: Show HN: Rocky – Rust SQL engine with branches, replay, column lineage

#35

Post by hugocorreia90 »

thelittlenag wrote: post #31
Xiaoher-C wrote: post #23

The compile-time lineage part is the most interesting bit to me. A lot of “data lineage” tools feel like archaeology after the fact: parse logs, reconstruct what probably happened, then hope it matches reality. Having the compiler know “this column flows into these downstream models” before execution changes the workflow quite a bit. It makes refactors and masking policies much less scary. Do you expose any kind of “…

Same. I worked on an in-house product many years ago now where lineage and provenance were the entire point. Really cool to see this!

Thank you!

Re: Show HN: Rocky – Rust SQL engine with branches, replay, column lineage

#36

Post by hugocorreia90 »

Xiaoher-C wrote: post #23

The compile-time lineage part is the most interesting bit to me. A lot of “data lineage” tools feel like archaeology after the fact: parse logs, reconstruct what probably happened, then hope it matches reality. Having the compiler know “this column flows into these downstream models” before execution changes the workflow quite a bit. It makes refactors and masking policies much less scary. Do you expose any kind of “…

Hey Xiaoher-C! Hum, I don't have a lineage diff command yet. As of now, I can make a small lift to wire up together two commands that already exist: "rocky ci --diff --base main" which runs a diff between main and HEAD, and "rocky lineage --column columnname --downstream". I'll add this to my backlog!

Re: Show HN: Rocky – Rust SQL engine with branches, replay, column lineage

#37

Post by data_ders »

hugocorreia90 wrote: post #33
data_ders wrote: post #24

hiya, anders from dbt here. cool project -- I especially love the branching and budgeting options you've built in. both are things that I'd love for the dbt standard to include one day. was it dbt's lack of those feature that inspired you to start this project? It also seems you have an aversion to Jinja, which, believe me, I get! FYI dbt-fusion [1] is going GA next week (though GA for Databricks will come later) Mos…

Hey Anders! Thanks a lot for dropping a comment and show interest in Rocky. Yes! I won't going to lie that Jinja is one of the things that gives me some itches :). But it wasn't the major reason for start building Rocky though. It all started with the need for auto-generating dbt models from the FiveTran connections I integrate with, then having to hot reload code location in Dagster to discover new assets. All in a…

thanks for the context!

> Auto-generating these dbt models and get the manifest aligned between Dagster code location

I just added you on LinkedIn. if you accept my connection there I can DM you a private preview document that you might find very interesting related to dbt project metadata (that is way less painful than `manifest.json`)

Re: Show HN: Rocky – Rust SQL engine with branches, replay, column lineage

#38

Post by hugocorreia90 »

data_ders wrote: post #37
hugocorreia90 wrote: post #33

Earlier quoted context omitted.

Hey Anders! Thanks a lot for dropping a comment and show interest in Rocky. Yes! I won't going to lie that Jinja is one of the things that gives me some itches :). But it wasn't the major reason for start building Rocky though. It all started with the need for auto-generating dbt models from the FiveTran connections I integrate with, then having to hot reload code location in Dagster to discover new assets. All in a…

thanks for the context! > Auto-generating these dbt models and get the manifest aligned between Dagster code location I just added you on LinkedIn. if you accept my connection there I can DM you a private preview document that you might find very interesting related to dbt project metadata (that is way less painful than `manifest.json`)

Accepted! :)

Re: Show HN: Rocky – Rust SQL engine with branches, replay, column lineage

#39

Post by onlyrealcuzzo »

hugocorreia90 wrote: post #9
mollerhoj wrote: post #6

Its a bit confusing to claim that "The things your current stack can't give you because it doesn't own the DAG" and use DataBricks as your example: DataBricks includes jobs and pipelines, so it very much owns the DAG, no?

Fair point. Databricks owns a scheduling DAG (Workflows, DLT). What I meant by "owns the DAG" is the semantic DAG: model-to-model dependencies with column-level types that the compiler builds. Workflows knows task A runs before task B. Rocky knows `dim_customer.email` flows from `raw_users.email_address` through three CTEs in `stg_customers`. Different layer, same word. I'll be more careful with that framing.

> I'll be more careful with that framing.

I think you should also try to do a better job selling the benefit of this.

As a data engineer, I can see why this might be useful, but glancing through your README, the dots were not completely connected

Re: Show HN: Rocky – Rust SQL engine with branches, replay, column lineage

#40

Post by ModernMech »

FrustratedMonky wrote: post #27
ModernMech wrote: post #20

Earlier quoted context omitted.

It's really not hard to tell. It's the "How do you do fellow kids" of AI-isms. The presence of "fair pushback" and a single em dash reads as 99% AI generated as far as I am concerned. Yes, if you don't want to sound like you're cargo culting AI, you do have to change the way you talk because people aren't going to care otherwise. At the very least just because it's boring. That's always been the nature of slang and l…

"not hard to tell" Or, with all of the AI slop, you think you are detecting all AI. And don't realize the stuff that is AI and not noticed. There is a wide variety of tools now, with different degrees of output quality. https://ifunny.co/picture/it-s-been-forever-it-s-been-foreve...

I'm fine with work that uses AI. I use AI every day. I'm not fine with AI slop and it's very easy to tell what is slop and what's not, the same way it's easy to distinguish a selfie from a museum quality photograph. Are some selfies works of art? Few and far between, so you'd be forgiven if you dismiss all selfie-looking photographs as not worth your time.
Post reply on HN