Word. Most companies don't have nearly enough data to begin to justify them.
Most companies do not need Snowflake or Databricks
21–30 of 140 posts
Re: Most companies do not need Snowflake or Databricks
#22True from a purely technical point-of-view, but doesn't take into account how companies make decisions about adopting platform tech and that they're making these choices for good reasons.
When companies choose SF or DB they don't make a decision to install one of these platforms because there is no combination of open-source components and bespoke engineering work that could replace them and even be more flexible and efficient. They are choosing these platforms because they make it easy to manage data operations holistically, because it's easier to hire people who already know them well, because they are opinionated and restrict the range of bad or just strange decisions someone in the organisation could be making if they were not restricted, and of course for enterprise support, available from a single reputable vendor.
See also: 80% of companies don't need Kubernetes, 80% of companies don't need a big-3 cloud, 80% of companies don't need managed security solutions, 80% of companies don't need SAP, 80% of companies don't need Salesforce, etc etc etc...
Re: Most companies do not need Snowflake or Databricks
#23I would turn this on it’s head and say that most companies do need one or the other.
My first premise is that most companies will have a BI or Data and Analytics problem, whether it’s analysing their spend, revenue, operations, customer churn or something more interesting.
At that point, having an industry standard, fully managed, fully elastic and resilient platform with consumption based pricing sounds pretty appealing.
Yes I can run and administer a warehouse on EC2, but the total cost of manpower and servers with full resilience is going to be high, especially as you’ll have to add in analytics tools, ETL tools etc which Snowflake or Databricks might have included.
I’m a huge believer in both Snowflake and Databricks. Snowflake for BI and Databricks for anything more funky. The technology is on point and the business case stacks up for the most part.
Re: Most companies do not need Snowflake or Databricks
#24Managing data yourself is hard. It is hard not because it can't scale, but for more boring reasons like harder DB upgrades and migrations, snapshotting, access control, lacking libraries/SDK, lack of documentation, harder training of new employees etc. The number one reason for extended downtime I have seen in companies is that data is in some bad state. And good data engineers who could do all the things authors exp…
Yeah. This post reads like the Dropbox comment.
I've worked with ad-hoc data solutions, and now I work with analysts who swear by Snowflake. It's the organization, the administration, the maintenance that Snowflake is solving for us, and it's turn-key. It's not easy or cheap to get that done with a local solution.
Re: Most companies do not need Snowflake or Databricks
#2599.99% of companies are not Google sized and don't need Google's solutions. Period.
Google-sized solutions tend to create the need for google sized solutions though.
Take a basic CRUD TODO app. Add metrics to every user interaction, every mouse movement. Now you need a time series database, kafka and a data lake. Break it down into microservices. Now you need kubernetes, and also structured logging, and probably an ELK stack. You also probably need Graphana to monitor your services. Because you have introduced hundreds of failure points, you also need high-availability, multiple regions, elastic scaling. Eventual consistency.
Re: Most companies do not need Snowflake or Databricks
#26Agreed, not completely on duckdb but we used it for consolidating billing data from 10+ ERP systems and it works, so I see his point. Just to add to his points: - Integrations are still one of the hardest things in Enterprise IT. Snowflake/Databricks/etc in fact add to the number of systems to integrate, they make this problem worst most of the times - Governance in a self-service data ecosystem gets complicated fast…
I guess the ETL orchestration was the hard part? How did you do it?
Connectivity was the hardest part, we had to write Python connectors for a variety of ERP systems. We have a 1 server setup where we run duckdb and Python scripts, we monitor and orchestrate with Prefect but you can use one of the many Python orchestration tools. We load the finished data marts to a MS SQL server and users connect to it via PowerBI or Excel.
Re: Most companies do not need Snowflake or Databricks
#27True from a purely technical point-of-view, but doesn't take into account how companies make decisions about adopting platform tech and that they're making these choices for good reasons. When companies choose SF or DB they don't make a decision to install one of these platforms because there is no combination of open-source components and bespoke engineering work that could replace them and even be more flexible and…
Many companies also install DB, k8, etc. because costs don't matter as much. In the times of easy money they focus on running fast, borrowing if needed.
When money is harder to borrow, as it is now, some of those decisions get reassessed. My 2c.
Re: Most companies do not need Snowflake or Databricks
#28I work as a solution architect at a consulting firm that builds analytical data platforms for customers. Our company has a partnership with Snowflake, which means all the solutions we build are pushed to use Snowflake. Their sales strategy is very Oracle-like and at least in my circles many Snowflake sales employees are ex-Oracle. This means our sales and Snowflake sales are the best of friends. Formally they'll deny…
> and at least in my circles many Snowflake sales employees are ex-Oracle.
given my previous experiences with Snowflake sales, this would not surprise me in the slightest. A small list of the incredibly pushy things SF sales teams did:
- called up junior team members and pushed them to sign a renewal deal - point blank refused to explain how credits translated to real world money when pressed about how much usage we were actually getting for our money - after being told not to, they continued to call team members and pressured them to commit to a renewal - TAC/support gave consistently vague and unhelpful explanations.
So yeah, Snowflake is an awful company and an overpriced product, and I hope it fails.
Re: Most companies do not need Snowflake or Databricks
#29I work as a solution architect at a consulting firm that builds analytical data platforms for customers. Our company has a partnership with Snowflake, which means all the solutions we build are pushed to use Snowflake. Their sales strategy is very Oracle-like and at least in my circles many Snowflake sales employees are ex-Oracle. This means our sales and Snowflake sales are the best of friends. Formally they'll deny…
The heart of Snowflake is a very solid data warehouse, and one with very few configuration options.
Yes there other bells and whistles, but it doesn’t feel as though you are buying something hugely overblown even if you are doing something very vanilla.
And as you say, it’s all consumption based pricing so I think it’s unfair to characterise their sales team as pushing the wrong solution into the market.
Re: Most companies do not need Snowflake or Databricks
#30...a talk with your local Dev Ops Engineer/ manager to discuss how to secure your implementations
This right here is exactly why Snowflake is a good fit for my org.: we could pay someone's salary to install, maintain, and upgrade some open source alternative (and the VPS to run it), or we could just pay for Snowflake and stop wasting engineering's time on the Data Science's team's stuff, which frees them up to move faster on the core platform features bringing in the big bucks.
This reads like a flavor of the same argument I've heard many times over. You know what the biggest cost of an org. really is as we head into a recession? Labor. Stop wasting it on stuff done better by specialized companies that free you up to get stuff done.