Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Rocky – Rust SQL engine with branches, replay, column lineage

github.com

Post reply on HN
Print view View original

41–50 of 58 posts

Re: Show HN: Rocky – Rust SQL engine with branches, replay, column lineage

#41

Post by jtagliabuetooso »

Xiaoher-C wrote: post #23

The compile-time lineage part is the most interesting bit to me. A lot of “data lineage” tools feel like archaeology after the fact: parse logs, reconstruct what probably happened, then hope it matches reality. Having the compiler know “this column flows into these downstream models” before execution changes the workflow quite a bit. It makes refactors and masking policies much less scary. Do you expose any kind of “…

Data contracts as types and compile time checks (even across languages) are not new - this is a recent paper exposing the idea of correctness-by-design pipeline, which is a super set of this particular issue obviously (disclaimer: I'm one of the author of the paper): https://arxiv.org/pdf/2602.02335

Re: Show HN: Rocky – Rust SQL engine with branches, replay, column lineage

#42

Post by hugocorreia90 »

hobofan wrote: post #32
PeterWhittaker wrote: post #17

Congrats on the work, but have you considered another name? Naming is hard and always will be: When I first scanned the headline, my initial thought was "that's an interesting area for the Rocky Linux team to explore". After a moment, "wait, no, that's confusing, it's some other Rocky".

I fear that there is an even closer candidate for confusion: RocksDB

Oh yeah, good call.

Re: Show HN: Rocky – Rust SQL engine with branches, replay, column lineage

#43

Post by hugocorreia90 »

onlyrealcuzzo wrote: post #39
hugocorreia90 wrote: post #9

Earlier quoted context omitted.

Fair point. Databricks owns a scheduling DAG (Workflows, DLT). What I meant by "owns the DAG" is the semantic DAG: model-to-model dependencies with column-level types that the compiler builds. Workflows knows task A runs before task B. Rocky knows `dim_customer.email` flows from `raw_users.email_address` through three CTEs in `stg_customers`. Different layer, same word. I'll be more careful with that framing.

> I'll be more careful with that framing. I think you should also try to do a better job selling the benefit of this. As a data engineer, I can see why this might be useful, but glancing through your README, the dots were not completely connected

Make sense. Reviewing the README is on my TODO list. Thanks for the heads up!

Re: Show HN: Rocky – Rust SQL engine with branches, replay, column lineage

#44

Post by jtagliabuetooso »

Cool release.

IMO, "Why it's distinctive" is a bit misleading on a few points: certainly the dbt and DX folks can add their POV, but even considering stuff I know / authored ;-), https://arxiv.org/pdf/2308.05368 from 2023 (and following releases) cover branches in a native way (no clone), immutability (re-run), and lineage.

Extensions to be considered are different languages (what about Python), and branch semantics. Two immediate questions would be: can you nest branches? How does merge works across systems if you don't control compute?

Re: Show HN: Rocky – Rust SQL engine with branches, replay, column lineage

#45

Post by hugocorreia90 »

PeterWhittaker wrote: post #26
hugocorreia90 wrote: post #22

Earlier quoted context omitted.

Thanks Peter. All my side-projects are named after my pets. I had a dog named Rocky and given this project is also an underdog competing with well-established tools such as dbt and sqlmesh, I decided to keep Rocky when opening it to public. But I'm happy to get some suggestions for a better name to this tool :)

I love that! I am inspired to create Terry, Tizzie, Topé, Bubba, and Roxy (the three Ts are in my office right now), the last two are no longer with us but for the hole in my heart. I have no idea what these projects would be, but based on personalities, Roxy would chew through CPU and memory like a beaver (she loved turning large branches into small chunks), Bubba would inspire calm and peacefulness but walk into th…

Love your pet names and how you characterize them if you would name something after them :) Wiley is an interesting name!

I have another side-project, still private, which I named Shimi, my current dog's name. I'd thought naming my dog Sashimi, but Shimi is just shorter and simpler. I'm now considering stealing the name from my this other side-project for renaming Rocky, but I'll put more thought into it :)

Re: Show HN: Rocky – Rust SQL engine with branches, replay, column lineage

#47

Post by hugocorreia90 »

jtagliabuetooso wrote: post #41
Xiaoher-C wrote: post #23

The compile-time lineage part is the most interesting bit to me. A lot of “data lineage” tools feel like archaeology after the fact: parse logs, reconstruct what probably happened, then hope it matches reality. Having the compiler know “this column flows into these downstream models” before execution changes the workflow quite a bit. It makes refactors and masking policies much less scary. Do you expose any kind of “…

Data contracts as types and compile time checks (even across languages) are not new - this is a recent paper exposing the idea of correctness-by-design pipeline, which is a super set of this particular issue obviously (disclaimer: I'm one of the author of the paper): https://arxiv.org/pdf/2602.02335

Hey jtagliabuetooso! Absolutely, the idea isn't new. Rocky's bet is on the shipped implementation. I'll read your paper, thank you for sharing.

Re: Show HN: Rocky – Rust SQL engine with branches, replay, column lineage

#48

Post by hugocorreia90 »

jtagliabuetooso wrote: post #44

Cool release. IMO, "Why it's distinctive" is a bit misleading on a few points: certainly the dbt and DX folks can add their POV, but even considering stuff I know / authored ;-), https://arxiv.org/pdf/2308.05368 from 2023 (and following releases) cover branches in a native way (no clone), immutability (re-run), and lineage. Extensions to be considered are different languages (what about Python), and branch semantics.…

Hey jtagliabuetooso! Rocky is SQL-first instead of Python-first, and that was a deliberate scope choice. Also, Rocky acts as control plane that delegates compute to the warehouse (Databricks, etc) rather than owning the runtime. I don't have a strong use-case for nested branches, so no, that's not a feature Rocky has of today. I'll read the paper, thank you for sharing.

Re: Show HN: Rocky – Rust SQL engine with branches, replay, column lineage

#49

Post by jtagliabuetooso »

hugocorreia90 wrote: post #47
jtagliabuetooso wrote: post #41

Earlier quoted context omitted.

Data contracts as types and compile time checks (even across languages) are not new - this is a recent paper exposing the idea of correctness-by-design pipeline, which is a super set of this particular issue obviously (disclaimer: I'm one of the author of the paper): https://arxiv.org/pdf/2602.02335

Hey jtagliabuetooso! Absolutely, the idea isn't new. Rocky's bet is on the shipped implementation. I'll read your paper, thank you for sharing.

“Why it’s distinctive” is misleading (perhaps LLM-generated)?

Imo, we cite other work because it puts our work in context for experts and beginners alike; because it makes clear that we all stand on someone else’s shoulders (progress is, most of the time, a collective endeavor, not a lone-genius affair); because it is intellectually honest to acknowledge our debts.

Especially today, when putting research ideas out there almost guarantees they will be plagiarized by someone vibe-coding or vibe-writing, recognizing that our contributions come from somewhere is more important than ever. The implementation may or may not be novel, but the fact that it depends on LLMs even in the README should make you even more aware of why proper attribution is crucial: what's the incentive for open innovation if we all behave like this?

Re: Show HN: Rocky – Rust SQL engine with branches, replay, column lineage

#50

Post by jtagliabuetooso »

hugocorreia90 wrote: post #48
jtagliabuetooso wrote: post #44

Cool release. IMO, "Why it's distinctive" is a bit misleading on a few points: certainly the dbt and DX folks can add their POV, but even considering stuff I know / authored ;-), https://arxiv.org/pdf/2308.05368 from 2023 (and following releases) cover branches in a native way (no clone), immutability (re-run), and lineage. Extensions to be considered are different languages (what about Python), and branch semantics.…

Hey jtagliabuetooso! Rocky is SQL-first instead of Python-first, and that was a deliberate scope choice. Also, Rocky acts as control plane that delegates compute to the warehouse (Databricks, etc) rather than owning the runtime. I don't have a strong use-case for nested branches, so no, that's not a feature Rocky has of today. I'll read the paper, thank you for sharing.

How is the Git semantics of merge, rebase and diff defined in the system?
Post reply on HN