No one needs to evaluate all options against their current/known/possible needs when they can just pick a well vetted product and get on with business.
Most companies do not need Snowflake or Databricks
61–70 of 140 posts
Re: Most companies do not need Snowflake or Databricks
#62As a data engineer who's worked in a bunch of different contexts/companies I 100% agree most of the time snowflake/databricks is an unnecessary money sink. The main problem is most companies need the security of a managed service for cloud computing, and don't want to be locked out of scaling to a very large scale (with distributed compute). Unfortunately, I don't think there are a lot of options that meet those pret…
I think your second point is vastly overstated for 99% of companies and their use cases. There are precious few situations that can't be handled decently well by a simple RDBMS, and a simple ETL process into an analytical data store for reporting. Almost everything that exist fits one of three buckets: 1) CRUD app, 2) reporting tool, 3) business process app. And almost all of this is for internal use only. It's reall…
I think you're right and I probably came over a bit strong in how likely I made out companies would be to use that level of scale. The point I was trying to make was more that because using something like RDBMS (maybe + VMs or something for data science) has a hard ceiling in terms of scale, even if companies are really unlikely to hit that ceiling, there is a weird incentive structure.
The costs of databricks or snowflake are really high, but probably not enough to bankrupt a company or division. If you're a small to midsize company and you realise you can't compute on your data, that could cause much more serious issues. Even if those issues are only feasible 1/100 times, it's a pretty scary risk to expect a CTO to ignore.
Re: Most companies do not need Snowflake or Databricks
#63> The global economy is headed for a recession. That’s not my opinion, that’s the Federal Reserve’s
I know that's not the topic of the post but OP shows a lack of comprehension here. The Fed warned because that's what they do when the risk is non-negligible. They never say it is definitely going to happen because nobody knows.
Re: Most companies do not need Snowflake or Databricks
#64[deleted]
Re: Most companies do not need Snowflake or Databricks
#65...a talk with your local Dev Ops Engineer/ manager to discuss how to secure your implementations This right here is exactly why Snowflake is a good fit for my org.: we could pay someone's salary to install, maintain, and upgrade some open source alternative (and the VPS to run it), or we could just pay for Snowflake and stop wasting engineering's time on the Data Science's team's stuff, which frees them up to move f…
>This right here is exactly why Snowflake is a good fit for my org.: we could pay someone's salary to install, maintain, and upgrade some open source alternative (and the VPS to run it), or we could just pay for Snowflake and stop wasting engineering's time on the Data Science's team's stuff, which frees them up to move faster on the core platform features bringing in the big bucks.
The cost problem really is out of control. You'd come out ahead financially pretty quickly with a DIY solution. DBUs are on the order of ~100x the actual underlying compute cost. I simply cannot understand how anyone chooses this over running your own Spark clusters with Jupyterlab. There are single-click image deploys that do this stuff.
Re: Most companies do not need Snowflake or Databricks
#66Most companies don't need a lot of things. I don't do a whole lot of data analytics anymore, but I'd say: Start with figuring out how much data you actually have. We still see companies claim to have vast amounts of data, but in reality they are talking about less than a TB of data, frequently just a few 100GB. When you operate at that scale, just chuck your data into whatever database you're comfortable with and do…
A Terabyte of data can be still vast, depending on how different and unstructured they are. If you have thousands of different logfiles-formats, datadump-formats and various complex fileformats, then you need a very different solution than you would need to handle a terabyte of uniform data in a csv-file or similar.
Spark wont inherently help you solve that problem though, and neither will most of the big data solutions.
Re: Most companies do not need Snowflake or Databricks
#67Earlier quoted context omitted.
>open source it may be. free it is not. paying an expert to correctly deploy an open source solution takes time and money. oh you want it maintained? Funny. I heard this same kind of argument used against replacing oracle with postgres. It reminds me of Microsoft's "TCO" PR offensive back when they were public about how much they loathed open source competition. Thing is that it wasnt just a straw man (nobody that ne…
context matters. you’re calling it an organization, that alone typically indicates a larger scale, in which supporting something internal might be feasible. or at least it is a term used by those that have been there. you’re also referencing politics, which, again, that’s highly suggestive of a specific type of experience. experience within scale if not at scale. please don’t mistake my intent to be that open source…
I think you are vastly under-estimating the amount of cost/manpower needed for snowflake and cloud based solutions (they are anything but turn-key and have a lot of churn). There are cases for both but it's not as simple as open source = hire and cloud = no people needed.
With Snowflake you are paying for not only the development of the product, the hosting of it, the engineers to run it, but also the sales, marketing, and management behind it. But on top of all that you probably need to hire people to implement and maintain a solution utilizing it for your organization.
Re: Most companies do not need Snowflake or Databricks
#68i am not sure why everyone (comments, this post, etc.) assume there is a one-size-fits-all solution to every problem, even this one that looks quite simple. companies have to align a few things, ranging from the skills of the current employees, hiring plans, investor/shareholder management, or how to make sure the CEO really gets that boat he really wants and deserves.
not all businesses are the same. businesses with fat contracts but few users won't have massive operating costs, and they can use whichever easy and non-scalable technology they want, because once the business scales up, those fat contracts will pay for enough data engineers. a gaming startup will face high server costs right away, without any optimisation, while data platforms (e.g. bigquery) with a tiny bit of optimisation (materialising 2-3 summary tables, for example) will bring the cost down to "laughable" pretty easily.
it is true that many of these things are choices, e.g. do you really want to spend a shit ton of money for looker when superset for most users is just as good? are you even able to make that choice? if these choices are hard to make because a potential user (or set of users) in the company really wants something instead of something else, well, that is not a technical choice, and the issue you have has nothing to do with the technology.
Re: Most companies do not need Snowflake or Databricks
#69...a talk with your local Dev Ops Engineer/ manager to discuss how to secure your implementations This right here is exactly why Snowflake is a good fit for my org.: we could pay someone's salary to install, maintain, and upgrade some open source alternative (and the VPS to run it), or we could just pay for Snowflake and stop wasting engineering's time on the Data Science's team's stuff, which frees them up to move f…
>This right here is exactly why Snowflake is a good fit for my org.: we could pay someone's salary to install, maintain, and upgrade some open source alternative (and the VPS to run it), or we could just pay for Snowflake and stop wasting engineering's time on the Data Science's team's stuff, which frees them up to move faster on the core platform features bringing in the big bucks. The cost problem really is out of…
Here was my situation. Occasional queries. Over a couple petabyte of data. Customer facing so response in seconds per SLA but >
95 percent of the time the warehouse isn’t running. Cached queries from within 24 hours which don’t require the warehouse to even spin up. Our snowflake costs were significantly less than an FTE.
Would that potentially be a situation which “running your own” doesn’t make sense?
Re: Most companies do not need Snowflake or Databricks
#70I think most companies don't understand a large part of the Databricks offering and it should be used by way more organizations. Disclaimer: I was a Databricks user for 6 years and now work at Databricks.
Yea, you can create your own Spark deployment, but it will run much slower than the Databricks Runtime (DBR) or the Databricks proprietary Spark Runtime (Photon). Computations that run slower cause you to have a larger cloud compute bill. Databricks rewrote Spark in C++ and it runs really fast and saves a lot on ec2 compute.
> Define when you should compact files, when to Z-order
Or don't consider these issues and use autocompaction / the new Liquid clustering. These are great examples of problems the platform should solve, so the user has time to focus on business logic.
> If you can sniff out the inefficiencies in your Data early and make architecture that handles your specific data
I don't know what this means.
Are you going to build a deep learning model to make read/writes faster like Databricks predictive I/O? https://docs.databricks.com/en/optimizations/predictive-io.h.... Probably not, you have a lot of business problems to solve.
> Do the real work. Work with people. The Code will write itself.
I've seen lots of DIY data platforms. They're horrible to work with and I can assure you that the code does not write itself. The data engineers have a lot less time to write code because they're constantly trying to stand the platform back up.