Snowflake Metadata powered by FoundationDB

#1

Post by ashishnm » Thu, Apr 19, 2018, 9:36 PM UTC

Snowflake Metadata powered by FoundationDB
snowflake.net

Re: Snowflake Metadata powered by FoundationDB

#2

Post by bithavoc » Fri, Apr 20, 2018, 1:39 AM UTC

So when FoundationDB was acquired, Snowflake kept using it as some sort of internal fork? Wow

Re: Snowflake Metadata powered by FoundationDB

#3

Post by jitl » Fri, Apr 20, 2018, 1:58 AM UTC

bithavoc wrote: post #2

So when FoundationDB was acquired, Snowflake kept using it as some sort of internal fork? Wow

Yeah, I’m curious to know how this worked.

Re: Snowflake Metadata powered by FoundationDB

#4

Post by manigandham » Fri, Apr 20, 2018, 3:02 AM UTC

Will any of these layers on top be open-sourced? FDB itself is rather low-level and will result in duplicate efforts building another SQL/data layer.

It's rather interesting to see how many companies have been using FoundationDB silently this whole time...

Re: Snowflake Metadata powered by FoundationDB

#5

Post by newsat13 » Fri, Apr 20, 2018, 3:27 AM UTC

bithavoc wrote: post #2

So when FoundationDB was acquired, Snowflake kept using it as some sort of internal fork? Wow

not a fork. they got it via an escrow

Re: Snowflake Metadata powered by FoundationDB

#6

Post by ashishnm » Fri, Apr 20, 2018, 5:02 AM UTC

bithavoc wrote: post #2

So when FoundationDB was acquired, Snowflake kept using it as some sort of internal fork? Wow

Yes, we kept using FoundationDB, primarily because of its reliability. The layers we built on top don't have knowledge of the underlying architecture.

Re: Snowflake Metadata powered by FoundationDB

#7

Post by ashishnm » Fri, Apr 20, 2018, 5:25 AM UTC

manigandham wrote: post #4

Will any of these layers on top be open-sourced? FDB itself is rather low-level and will result in duplicate efforts building another SQL/data layer. It's rather interesting to see how many companies have been using FoundationDB silently this whole time...

To be clear we haven't built a SQL layer on top of FDB. It's a metadata store for us where we store objects. We haven't decided whether to open source the object-mapping layer. It's plausible, and if we do it will take some work. But we are certain to contribute to the FDB community. So do keep an eye out.

Re: Snowflake Metadata powered by FoundationDB

#8

Post by thepumpkin1979 » Fri, Apr 20, 2018, 3:39 PM UTC

newsat13 wrote: post #5

bithavoc wrote: post #2

So when FoundationDB was acquired, Snowflake kept using it as some sort of internal fork? Wow

not a fork. they got it via an escrow

You’re right https://news.ycombinator.com/item?id=9262276

Re: Snowflake Metadata powered by FoundationDB

#9

Post by edwinyzh » Mon, Apr 23, 2018, 5:53 AM UTC

@ashishnm, Thanks for sharing, after I read the HN excitement around the open sourcing of foundationdb, I had a question in my mind and I hope it's not to off-topic to ask here - do you happened to know is it feasible to implement a SQL Layer on top of foundationdb, using Sqlite's Virtual Table mechanism (http://www.sqlite.org/vtab.html)?