Most companies do not need Snowflake or Databricks
41–50 of 140 posts
Re: Most companies do not need Snowflake or Databricks
#42Earlier quoted context omitted.
> and at least in my circles many Snowflake sales employees are ex-Oracle. given my previous experiences with Snowflake sales, this would not surprise me in the slightest. A small list of the incredibly pushy things SF sales teams did: - called up junior team members and pushed them to sign a renewal deal - point blank refused to explain how credits translated to real world money when pressed about how much usage we…
sorry, but, there’s something about this that doesn’t smell right. junior ic being asked to sign a contract makes zero sense. that’s insanity. plenty of managers at plenty of companies lack that level of authority. sales teams use slimy tactics. that’s a given. time limits, multi-years, etc, etc, etc. trying to get the wrong person to sign is a different category entirely. maybe you ran across the “best of the best”…
[deleted]
Re: Most companies do not need Snowflake or Databricks
#43Most companies don't need a lot of things. I don't do a whole lot of data analytics anymore, but I'd say: Start with figuring out how much data you actually have. We still see companies claim to have vast amounts of data, but in reality they are talking about less than a TB of data, frequently just a few 100GB. When you operate at that scale, just chuck your data into whatever database you're comfortable with and do…
> We still see companies claim to have vast amounts of data, but in reality they are talking about less than a TB of data, frequently just a few 100GB. When you operate at that scale, just chuck your data into whatever database you're comfortable with and do SQL queries, it's fine.
I did some work with a company that talked about their “big data”, bragged about their data lake, and made grandiose statements about how they had big plans to use “all this data” to get an edge up on their competitors. They talked about the big data platforms and technologies they were evaluating and how they’d probably have to hire more people to handle it all.
When I pressed them for details on it, they eventually explained the data they had on each customer and the approximate number of customers. I did some quick math and calculated the total amount of data on all of their customers was under 100MB.
They had managed to blow it up to occupy a couple orders of magnitude more space in one situation (think storing time series data as JSON where every point had duplicated tags attached due to ineffective structure) but even that was small enough to fit entirely in RAM of a decent sized server.
Re: Most companies do not need Snowflake or Databricks
#44Earlier quoted context omitted.
Any typical analytics use case, where a single dataset (think a single table) is less than 2TB and the total data size is less than 10-20 TB in the organization. If that's you, you very likely don't need Snowflake and definitely not Databricks. An RDBMS would suffice, I recommend Postgres. These sizes are just my thumb rules. Your mileage may vary. I have people in my network use Snowflake with a total data size of 5…
years ago, had this one college professor, students would spend the time before class asking him to divide two fractional numbers, and he would rattle off a few decimal places after calculating, to cheers from the students that had their calculators out with the answers. the excitement would build with every correct decimal point. he didn’t actually need a calculator. but he did own one. hopefully this does not cause…
Am I right in understanding that all these companies are actually making deals for on-demand use in the future?
Re: Most companies do not need Snowflake or Databricks
#45I work as a solution architect at a consulting firm that builds analytical data platforms for customers. Our company has a partnership with Snowflake, which means all the solutions we build are pushed to use Snowflake. Their sales strategy is very Oracle-like and at least in my circles many Snowflake sales employees are ex-Oracle. This means our sales and Snowflake sales are the best of friends. Formally they'll deny…
If you want something fresh and new in this space, check out FeatureBase: https://featurebase.com/. We just added vector/embeddings support to the cloud product.
Re: Most companies do not need Snowflake or Databricks
#46My client is in the middle of migrating their Snowflake data to Databricks Unity Catalog.
To this day, I still do not understand why they love Databricks so much. It just looks like a Jupyter notebook to me. I know, it has Spark. So?
I’m trying to learn more so I can build something better, maybe.
Re: Most companies do not need Snowflake or Databricks
#47> The cost for something that can be replicated free and open source is absurd. open source it may be. free it is not. paying an expert to correctly deploy an open source solution takes time and money. oh you want it maintained? the three recommendations sound like those of a consultant. they work great with exec buy-in, and are a joke without. yes, yes, you just have to explain yourself. just help management underst…
>open source it may be. free it is not. paying an expert to correctly deploy an open source solution takes time and money. oh you want it maintained?
Funny. I heard this same kind of argument used against replacing oracle with postgres. It reminds me of Microsoft's "TCO" PR offensive back when they were public about how much they loathed open source competition.
Thing is that it wasnt just a straw man (nobody that needs to be convinced is under any illusions that software has to be maintained), Oracle was also way more expensive to maintain ON top of being expensive to run.
They had their hooks into that organization pretty good though and once that happens technology choices become highly political - which experts are you going to fire and which ones are you going to hire?
The oracle lot obviously didnt want it to be them that get managed out.
Re: Most companies do not need Snowflake or Databricks
#48My experience is that there is a lot of milage for small and medium enterprises to use your normal RDBMS replication to create a copy of your OLTP DB that drives your business and run your analytics on that. And put up with complaining it can be slow to do inefficient analytics queries. Really, after spending silly amounts of time and hassle (and money) on the fancy snowflake or whatever, you discover its not massive…
I think "put up with complaining it can be slow to do inefficient analytics queries" is why I would rather pay for some 3rd party tool that would get the complaints.
Business people want some queries that are slow no matter what you do. Then expect these to run in under a second. They don't care about some abstract things like operations complexity or big O - they want it here and now.
Even if your data fits on one server there are still queries that will take time and it is not going to be instant, even if you put whole db in RAM it is going to be fast but still I can imagine bunch of queries that will simply have to take couple of minutes anyway.
Re: Most companies do not need Snowflake or Databricks
#49As a data engineer who's worked in a bunch of different contexts/companies I 100% agree most of the time snowflake/databricks is an unnecessary money sink. The main problem is most companies need the security of a managed service for cloud computing, and don't want to be locked out of scaling to a very large scale (with distributed compute). Unfortunately, I don't think there are a lot of options that meet those pretty simple requirements that aren't databricks or snowflake[1].
Sure, you could put your compute on single docker containers, but when your data gets too big your stuck and have to get someone expensive in to manage kubernetes, all that time, you can't compute on your data. Which is sorta the crux of it: databricks and snowflake are expensive, but not nearly as expensive as finding out you need them and not having them.
[1] if you're using python on was/gcp, I think coiled (coiled.io) is a rare exception to that pattern
Re: Most companies do not need Snowflake or Databricks
#50Considering what you get I think Snowflake is rather decently priced. It’s not easy to integrate and operate a bunch of open source tools that replaces it, especially not on a small scale. Also, in most cases you’ll pay a lot more for the people that use Snowflake than you do for Snowflake, so it makes more sense to focus on productivity.