Viewing profile — rxin

This is coming to Serverless SQL warehouses soon!

(OP / Databricks cofounder here) I'm not sure what happened but I just sent an email out internally to ask people not to do this. The team might have gotten overly excited by this …

(Databricks cofounder here) All datasets are biased, including this specific one. However, we believe it's still very valuable to open source, for a few reasons: - This dataset is …

Databricks founder here. Doesn’t seem like DM work on HN anymore. Do you mind shooting me an email? Would love to follow up and understand the issues. rxin@databricks.com

You should try Databricks, especially the new Photon engine powering Spark. In general more performant than Snowflake in SQL and a lot more flexible. (There are some cases in which…

Isn’t that what the official TPC does?

Totally. Simplicity is critical. That’s why we built Databricks SQL not based on Spark. As a matter of fact, we took the extreme approach of not allowing customers (or ourselves) t…

Geometric mean is commonly used in benchmarks when the workloads consists of queries that have large (often orders of magnitude) differences in runtime. Consider 4 queries. Two run…

Ah ok. Wasn't clear. I think some repro scripts will be available soon.

Check my reply, Leo.

Exactly. Not sure about Netflix special, but there are experts that have dedicated their professional careers to creating fair benchmarks. Snowflake should just participate in the …

There's an official TPC process to audit and review the benchmark process. This debate can be easiest settled by everybody participating in the official benchmark, like we (Databri…

Co-author of the paper here. I don't think your argument holds here at all. It's a common misconception to think high performance would require tight coupling of storage and query …

We have taken a very different approach with the Unity Catalog. It is designed opposite to the "cluster-centric" access control model, and will be user and role centric. Disclosure…

You should take a look at Databricks, in all three dimensions.

Disclaimer: I designed the system that won the 2014 GraySort based on Apache Spark, and the same system was extended by a different team to win the 2016 record in cloud sort. "Merg…