81–90 of 168 posts
Ive been following this and it’s kind of embarrassing to watch. I love working with Databricks and Snowflake. They both knock it out of the park for their respective use case. They’re amazing products. It makes no sense to fall out about this though. For a 100TB dataset with a funky calculation, Spark will trounce Snowflake. For a 1 row dataset, Snowflake will return before the spark job has been serialised.
What I find hilarious is that companies argue who can query 100 TB faster and try to sell this to people. I've been on the receiving end of offers by both of the companies in question and used both platforms (and sadly migrated some data jobs to them). While they can crunch large datasets, they are laughably slow for the datasets most people have. So while I did propose we use these solutions for our big-ish data pro…
Earlier quoted context omitted.
maybe clickhouse?
Clickhouse is good if you're building application. It has lot of great features and incredible performance, but there's an expectancy that people using it know what they're doing and can work around its limitations (like limited support for joins and sql in general). Something like Snowflake works much better when you're building a platform that you can give to two hundred data analysts or various skills spread over…
Earlier quoted context omitted.
Do you work for Databricks?
They must do. But if you've been in this area for long enough, I'd put my money on Databricks, if anything, because of their open source integrity
wow. it's been a while since we've had one of these!!! paging Oracle, MS SQL Server, SAP, Sybase, etc, etc....
I'm gonna make some popcorn and crack open a beer!!!
I really wish I could block all of Snowflake's domain from my inbox. Sadly, Google encourages spammers to just create a new email address. So I get a few emails each month from Snowflake who ask me to try their products. I've never done business with them and there's no unsubscribe link.
Fuck Snowflake for thinking it has any room to talk about integrity.
Earlier quoted context omitted.
Databricks is a shit platform that encourages terrible data practices and accretion of technical debt.
Seems pretty good to us! Can you give more information?
Great for proof of concepts, but when you start to build out complete pipelines please look into how to make the pipelines more sustainable and maintainable.
so, alternatives? Aside from the Azure/GCP/AWS internal offeringa I know about Snowflake and Firebolt, Databricks is new to me.
Redshift is pretty terrible, stay away. AWS is even worse at delivering promises than Databricks and that's saying something. I heard Google BigQuery is good. It is completely SaaS (like AWS Athena that works). Unicorns often run their own stack and you could replicate that, if you have the apetite. Netflix and Apple run Trino + Spark on k8s + Iceberg. Uber used their own Hudi thing, not sure if they still do.
What I find hilarious is that companies argue who can query 100 TB faster and try to sell this to people. I've been on the receiving end of offers by both of the companies in question and used both platforms (and sadly migrated some data jobs to them). While they can crunch large datasets, they are laughably slow for the datasets most people have. So while I did propose we use these solutions for our big-ish data pro…