Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Rocky – Rust SQL engine with branches, replay, column lineage

github.com

Post reply on HN
Print view View original

21–30 of 58 posts

Re: Show HN: Rocky – Rust SQL engine with branches, replay, column lineage

#21

Post by DoctorOW »

hugocorreia90 wrote: post #8
ramon156 wrote: post #7

If your introduction message already includes a bunch of uncurated claims and LLM smells, then what does that say about the code I'm about to run?

Yeah, fair pushback, and yes the intro was AI-assisted. Marketing is not my strength nor I am a native english speaker. I built this in about a month with heavy LLM tooling and the seed comment is part of that. I'm not going to pretend otherwise. The code is what it is. `cargo test --workspace` runs across 19 crates. CI on 5 platforms (macOS ARM/Intel, Linux x86/ARM, Windows). JSON output schemas are codegen-checked…

> I'd rather hear "the code is bad" than "the post sounds AI-written".

Of course you would. Reading through and judging the quality of AI output is the largest amount of effort in a world where you can get everything else by prompting. Please internalize this: If you want to be respected you will have to put in effort yourself. There is no way around this.

Re: Show HN: Rocky – Rust SQL engine with branches, replay, column lineage

#22

Post by hugocorreia90 »

PeterWhittaker wrote: post #17

Congrats on the work, but have you considered another name? Naming is hard and always will be: When I first scanned the headline, my initial thought was "that's an interesting area for the Rocky Linux team to explore". After a moment, "wait, no, that's confusing, it's some other Rocky".

Thanks Peter. All my side-projects are named after my pets. I had a dog named Rocky and given this project is also an underdog competing with well-established tools such as dbt and sqlmesh, I decided to keep Rocky when opening it to public. But I'm happy to get some suggestions for a better name to this tool :)

Re: Show HN: Rocky – Rust SQL engine with branches, replay, column lineage

#23

Post by Xiaoher-C »

The compile-time lineage part is the most interesting bit to me. A lot of “data lineage” tools feel like archaeology after the fact: parse logs, reconstruct what probably happened, then hope it matches reality.

Having the compiler know “this column flows into these downstream models” before execution changes the workflow quite a bit. It makes refactors and masking policies much less scary.

Do you expose any kind of “lineage diff” between branches? For example: this PR changes the downstream impact of `customer.email` from A/B/C to A/B/D. That would be useful in code review.

Re: Show HN: Rocky – Rust SQL engine with branches, replay, column lineage

#24

Post by data_ders »

hiya, anders from dbt here. cool project -- I especially love the branching and budgeting options you've built in. both are things that I'd love for the dbt standard to include one day. was it dbt's lack of those feature that inspired you to start this project? It also seems you have an aversion to Jinja, which, believe me, I get!

FYI dbt-fusion [1] is going GA next week (though GA for Databricks will come later) Most of it is source-available and ELv2-licensed, but there's a number of crates that are Apache 2.0, namely: dbt-xdbc, dbt-adapter, dbt-auth, dbt-jinja, dbt-agate. We also have plans to OSS more as time goes on (stay tuned).

I just wanted to call out the OSS crates in case you'd rather focus on "making your beer taste better" than have to re-build foundations. I'd love to hear if any of those crates come in handy for you (even more so if they don't work for you).

Feel free to reach out on LinkedIn or dbt community Slack if you ever want to chat more!

[1]: https://github.com/dbt-labs/dbt-fusion

Re: Show HN: Rocky – Rust SQL engine with branches, replay, column lineage

#25

Post by hugocorreia90 »

cmrdporcupine wrote: post #16
FrustratedMonky wrote: post #14

Earlier quoted context omitted.

Not sure why you are downvoted here. 'A-Lot' of side projects, hobby projects, etc.. are all using AI tools now. Also for marketing, every sales/marketing firm is using AI. So why critisize this guy inparticular. AI is pervasive, the train has left the station. So that is not a reason to criticize this project. There might be other reasons, I'm not sure, but not that an AI was used.

It's really a weird world now. I do think the author is doing a disservice to themselves by writing the post and comments using LLM, even if the code is mostly agent built. People can tell right away, all the LLM shibboleths are there... it feels cheap. Just write naturally and then Google translate, don't let the LLM speak on your behalf. What's going to distinguish projects that are built this way is the ability to…

I totally acknowledge that. The only reason for passing my replies through AI was just because it's my first time posting here and opening a side-project of mine publicly.

All the engine architecture decisions are mine though and this project came up to solve a real problem I currently have at work with a zero-touch data pipeline leveraging FiveTran, Dagster, dbt and Databricks. This is a data pipeline that servers multiple agencies and data producers who work with data from more than 300 clients and multiple connectors.

Rocky essentially was built based on all the time spent awaken at night thinking about all these problems and how could they be addressed differently, considering that dbt is not suiting well this particular use-case.

I decided to open Rocky to public for free because of two simple reasons: 1st is that it might help others and I fullfill my ego of having built something other people like and use. 2nd is that I'm the solo maintainer. A project can only get proper traction if more people contributes to it.

Re: Show HN: Rocky – Rust SQL engine with branches, replay, column lineage

#26

Post by PeterWhittaker »

hugocorreia90 wrote: post #22
PeterWhittaker wrote: post #17

Congrats on the work, but have you considered another name? Naming is hard and always will be: When I first scanned the headline, my initial thought was "that's an interesting area for the Rocky Linux team to explore". After a moment, "wait, no, that's confusing, it's some other Rocky".

Thanks Peter. All my side-projects are named after my pets. I had a dog named Rocky and given this project is also an underdog competing with well-established tools such as dbt and sqlmesh, I decided to keep Rocky when opening it to public. But I'm happy to get some suggestions for a better name to this tool :)

I love that! I am inspired to create Terry, Tizzie, Topé, Bubba, and Roxy (the three Ts are in my office right now), the last two are no longer with us but for the hole in my heart.

I have no idea what these projects would be, but based on personalities, Roxy would chew through CPU and memory like a beaver (she loved turning large branches into small chunks), Bubba would inspire calm and peacefulness but walk into things (he was one-eyed and a little clumsy), Terry would stick like glue (an eBPF program, maybe?), Tizzie would work well most of the time then destroy your stuff (an AI agent?), and Topé would always be there, but never quite willing to participate (a bad Windows driver?).

I don't the area well enough at all to suggest an alternate name, but maybe Wiley, which is an indirect reference to Dag from Barnyard via Wile E. Coyote?

Re: Show HN: Rocky – Rust SQL engine with branches, replay, column lineage

#27

Post by FrustratedMonky »

ModernMech wrote: post #20
FrustratedMonky wrote: post #19

Earlier quoted context omitted.

"Yeah, fair pushback" Really hard to tell. Because that used to be a common phrase that real people would use. So now I have to change my own language in order to not appear like I'm an AI? We are getting in a weird place where Humans have to act/sound increasingly 'odd', to appear not 'perfect' like an AI.

It's really not hard to tell. It's the "How do you do fellow kids" of AI-isms. The presence of "fair pushback" and a single em dash reads as 99% AI generated as far as I am concerned. Yes, if you don't want to sound like you're cargo culting AI, you do have to change the way you talk because people aren't going to care otherwise. At the very least just because it's boring. That's always been the nature of slang and l…

"not hard to tell"

Or, with all of the AI slop, you think you are detecting all AI. And don't realize the stuff that is AI and not noticed. There is a wide variety of tools now, with different degrees of output quality.

https://ifunny.co/picture/it-s-been-forever-it-s-been-foreve...

Post reply on HN