Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Melchi – Open-Source Snowflake to DuckDB Replication with CDC Support

github.com

Post reply on HN
Print view View original

11–18 of 18 posts

Re: Show HN: Melchi – Open-Source Snowflake to DuckDB Replication with CDC Support

#11

Post by ryanwaldorf »

chrisjc wrote: post #8
ryanwaldorf wrote: post #7

Earlier quoted context omitted.

Would that be to support SSO? If so, I haven't planned it yet but I could. Would that make this more useful to you?

It should be just as easy as adding: authenticator="externalbrowser" Adding a Snowflake connection configuration option that allowed for standard Snowflake connection conventions might be a good option. That way you could connect to Snowflake with your existing configurations (.snowsql/, .snowflake/). Or explicitly specify by adding matching args/params to your project's config. # myconf.toml [test-connection] accoun…

Thank you! Will take a look at this over the next few days

Re: Show HN: Melchi – Open-Source Snowflake to DuckDB Replication with CDC Support

#12

Post by ryanwaldorf »

chrisjc wrote: post #9
ryanwaldorf wrote: post #5

Earlier quoted context omitted.

That's really interesting! Could you tell me a bit more of what you're thinking? I'm not the most familiar with SQL Mesh and the typical workflows there.

Perhaps similar to https://github.com/duckdb/dbt-duckdb , but SQLMesh instead of DBT obviously.

Ah gotcha! Do you have a use case where you'd look to remodel/transform the data between warehouses?

Re: Show HN: Melchi – Open-Source Snowflake to DuckDB Replication with CDC Support

#13

Post by chrisjc »

ryanwaldorf wrote: post #12
chrisjc wrote: post #9

Earlier quoted context omitted.

Perhaps similar to https://github.com/duckdb/dbt-duckdb , but SQLMesh instead of DBT obviously.

Ah gotcha! Do you have a use case where you'd look to remodel/transform the data between warehouses?

Not the original parent, so unsure of their use-case. But I've seen the approach where some/basic development can be done on duckdb, before making its way to dev/qa/prd.

Something like your project might enable grabbing data (subsets of data) from a dev enviroment (seed) for offline, cheap (no SF warehouse cost) development/unit-testing, etc.

Re: Show HN: Melchi – Open-Source Snowflake to DuckDB Replication with CDC Support

#14

Post by ryanwaldorf »

chrisjc wrote: post #13
ryanwaldorf wrote: post #12

Earlier quoted context omitted.

Ah gotcha! Do you have a use case where you'd look to remodel/transform the data between warehouses?

Not the original parent, so unsure of their use-case. But I've seen the approach where some/basic development can be done on duckdb, before making its way to dev/qa/prd. Something like your project might enable grabbing data (subsets of data) from a dev enviroment (seed) for offline, cheap (no SF warehouse cost) development/unit-testing, etc.

This makes sense, thank you!

Re: Show HN: Melchi – Open-Source Snowflake to DuckDB Replication with CDC Support

#15

Post by pratio »

ryanwaldorf wrote: post #7
pratio wrote: post #6

This looks interesting, are there any plans to support snowflake external browser auth?

Would that be to support SSO? If so, I haven't planned it yet but I could. Would that make this more useful to you?

absolutely, without sso, I can't even try it. Our policies don't allow using a password.

Re: Show HN: Melchi – Open-Source Snowflake to DuckDB Replication with CDC Support

#16

Post by ryanwaldorf »

chrisjc wrote: post #8
ryanwaldorf wrote: post #7

Earlier quoted context omitted.

Would that be to support SSO? If so, I haven't planned it yet but I could. Would that make this more useful to you?

It should be just as easy as adding: authenticator="externalbrowser" Adding a Snowflake connection configuration option that allowed for standard Snowflake connection conventions might be a good option. That way you could connect to Snowflake with your existing configurations (.snowsql/, .snowflake/). Or explicitly specify by adding matching args/params to your project's config. # myconf.toml [test-connection] accoun…

I just pushed the external_browser branch to github that should offer 1/ externalbrowser authentication and 2/ the ability to use TOML files with profiles with instructions on how in the readme. if you run the following you should be able to test it out.

     git clone https://github.com/ryanwith/melchi.git
     git pull external_browser
     # remaining steps are the same
Would you mind testing it out and letting me know if it works for you? Would really appreciate it!

You can also let me know if it works in discord here: https://discord.gg/bTg9kJ92

Re: Show HN: Melchi – Open-Source Snowflake to DuckDB Replication with CDC Support

#17

Post by ryanwaldorf »

pratio wrote: post #15
ryanwaldorf wrote: post #7

Earlier quoted context omitted.

Would that be to support SSO? If so, I haven't planned it yet but I could. Would that make this more useful to you?

absolutely, without sso, I can't even try it. Our policies don't allow using a password.

just replied to chrisjc that I created a branch with externalbrowser auth and pushed it to github. would you mind taking a look and letting me know if it works for you? would love to get your feedback as I don't have SSO set up in my account to test this myself

Re: Show HN: Melchi – Open-Source Snowflake to DuckDB Replication with CDC Support

#18

Post by pratio »

ryanwaldorf wrote: post #17
pratio wrote: post #15

Earlier quoted context omitted.

absolutely, without sso, I can't even try it. Our policies don't allow using a password.

just replied to chrisjc that I created a branch with externalbrowser auth and pushed it to github. would you mind taking a look and letting me know if it works for you? would love to get your feedback as I don't have SSO set up in my account to test this myself

thank you, will try and let you know
Post reply on HN