99.99% of companies are not Google sized and don't need Google's solutions. Period.
Most companies do not need Snowflake or Databricks
11–20 of 140 posts
Re: Most companies do not need Snowflake or Databricks
#12We make a product that smaller companies use; it’s much simpler to use and no consultants needed. We notice that even in bigger corps, departments import into our system and use that because they find the de facto systems painful to use and they don’t actually need that much power, ever.
Re: Most companies do not need Snowflake or Databricks
#13Would be really wild if 20%, a truly whopping number, of the companies did need Showflake or Databricks.
I think the author means 80% of existing databricks/snowflake customers, since they say “the rest are overpaying”.
Re: Most companies do not need Snowflake or Databricks
#14For the non data experts, from elsewhere on medium:
“ Semi-structured data tends to evolve over time. Systems that generate data add new columns to accommodate additional information, which requires downstream tables to evolve accordingly.
The structure of tables in Snowflake can evolve automatically to support the structure of new data received from the data sources”
“When dealing with large datasets, the processing power of individual machines can become limiting, necessitating the use of distributed and parallel processing capabilities provided by platforms such as Databricks“
Re: Most companies do not need Snowflake or Databricks
#15heresy!
Re: Most companies do not need Snowflake or Databricks
#16Managing data yourself is hard. It is hard not because it can't scale, but for more boring reasons like harder DB upgrades and migrations, snapshotting, access control, lacking libraries/SDK, lack of documentation, harder training of new employees etc. The number one reason for extended downtime I have seen in companies is that data is in some bad state. And good data engineers who could do all the things authors expects them to do are not easy to find and expensive.
Re: Most companies do not need Snowflake or Databricks
#17Of course they do not need it. 80% of companies can also do without AWS, they "just" need to hire the people themselves, and developer a core competency running and administering those services. It doesn't automatically mean that it makes financial sense for said company. I guess "don't need" == "must not use" when you're selling yourself as filling that self-hosted gap.
This is one of the things I hate of this industry: service providers oversell their proposition and overcharge to infinity, while independent professionals often undersell that same proposition so that they can capture some of that overcharging for themselves. If you are an honest actor, you're left pulling your hair out at the overall mendacity of the whole market.
Profit can be such a cancer.
Re: Most companies do not need Snowflake or Databricks
#18Agreed, not completely on duckdb but we used it for consolidating billing data from 10+ ERP systems and it works, so I see his point. Just to add to his points: - Integrations are still one of the hardest things in Enterprise IT. Snowflake/Databricks/etc in fact add to the number of systems to integrate, they make this problem worst most of the times - Governance in a self-service data ecosystem gets complicated fast…
I guess the ETL orchestration was the hard part? How did you do it?
Re: Most companies do not need Snowflake or Databricks
#19I work as a solution architect at a consulting firm that builds analytical data platforms for customers. Our company has a partnership with Snowflake, which means all the solutions we build are pushed to use Snowflake. Their sales strategy is very Oracle-like and at least in my circles many Snowflake sales employees are ex-Oracle. This means our sales and Snowflake sales are the best of friends. Formally they'll deny kickbacks, but who knows?
For all my clients Snowflake is overkill when you look from the perspective of growth and scale. They'll never use that part of Snowflake. They might just do as well with DuckDB, Azure Synapse or any other analytical-oriented platform laying around.
What I do like with less-than-big use-cases is that (at least at Snowflake) you pay relatively little if you do relatively little data processing. It's not free, but it doesn't break the bank either.
Re: Most companies do not need Snowflake or Databricks
#20Which use cases cannot do without Snowflake or Databricks?
Any typical analytics use case, where a single dataset (think a single table) is less than 2TB and the total data size is less than 10-20 TB in the organization. If that's you, you very likely don't need Snowflake and definitely not Databricks. An RDBMS would suffice, I recommend Postgres.
These sizes are just my thumb rules. Your mileage may vary.
I have people in my network use Snowflake with a total data size of 50GB!!! It makes my blood boil.