Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Most companies do not need Snowflake or Databricks

kjhealey.medium.com

Post reply on HN
Print view View original

111–120 of 140 posts

Re: Most companies do not need Snowflake or Databricks

#111

Post by skinner927 »

sseagull wrote: post #59
mrweasel wrote: post #34

Most companies don't need a lot of things. I don't do a whole lot of data analytics anymore, but I'd say: Start with figuring out how much data you actually have. We still see companies claim to have vast amounts of data, but in reality they are talking about less than a TB of data, frequently just a few 100GB. When you operate at that scale, just chuck your data into whatever database you're comfortable with and do…

I find it frustrating to search for stuff for mid-scale problems sometimes. There's a lack of context about what 'metrics' or 'analytics' means, and that leads to over-engineered solutions. I run 2-3 (primarily database & web) servers. I thought it would be a nice idea to aggregate web accesses and stuff into a central location for analysis. It's probably 50-100MB of logs a day at most. But all the 'industry standard…

Fairly confident anyone outside of sales is going to tell you the solution that works is the best solution.

Re: Most companies do not need Snowflake or Databricks

#112

Post by ramesh31 »

ironchef wrote: post #69
ramesh31 wrote: post #65

Earlier quoted context omitted.

>This right here is exactly why Snowflake is a good fit for my org.: we could pay someone's salary to install, maintain, and upgrade some open source alternative (and the VPS to run it), or we could just pay for Snowflake and stop wasting engineering's time on the Data Science's team's stuff, which frees them up to move faster on the core platform features bringing in the big bucks. The cost problem really is out of…

Here was my situation. Occasional queries. Over a couple petabyte of data. Customer facing so response in seconds per SLA but > 95 percent of the time the warehouse isn’t running. Cached queries from within 24 hours which don’t require the warehouse to even spin up. Our snowflake costs were significantly less than an FTE. Would that potentially be a situation which “running your own” doesn’t make sense?

>Would that potentially be a situation which “running your own” doesn’t make sense?

Look into datalake architectures. RDBMS based data warehousing is obviously not economical at the petabyte scale. But storing all that data in S3 with Delta Lake/Iceberg format and querying with Spark changes things entirely. You only pay for object storage, and S3 read costs are trivial.

Re: Most companies do not need Snowflake or Databricks

#113

Post by rglullis »

hodgesrm wrote: post #107
rglullis wrote: post #60

Earlier quoted context omitted.

> we could pay someone's salary to install, maintain, and upgrade some open source alternative (and the VPS to run it), or we could just pay for Snowflake and stop wasting engineering's time You are missing the "we could pay someone that provides a commodified, open-source alternative to Snowflake" option. It would be cheaper, it wouldn't require any of your engineering team resources and it would help us all get rid…

> You are missing the "we could pay someone that provides a commodified, open-source alternative to Snowflake" option. Which services provider do you have in mind to pay? The advantage of Snowflake is that the CIO can pick Snowflake as the default for the entire org and it will work across a very large number of business use cases. I don't see any open source analytic software that can do that. In my experience open…

> The advantage of Snowflake is that the CIO can pick Snowflake as the default for the entire org

We are talking about companies that do not even have a CIO, which is just another way to say "most companies".

Re: Most companies do not need Snowflake or Databricks

#114

Post by pradeepchhetri »

I would consider following things before selecting a database offering:

- Whether the database vendor is a lock-in. It would be a straight "NO" for me if the database isn't open-source with proper license, since I can't self-host it in case, I need to move away from their SaaS offering due to various unexpected foreseen reasons: vendor decided to increase pricing, vendor has hidden pricing, vendor has reliability issues etc etc.

- How big is the community behind the database. Check their public forums to understand how community feels about the database and how their requirements are considered.

- Don't believe any random benchmarking post online but by doing my own benchmarking for my use-case.

- Check which other companies have adopted that database and read their experiences.

Re: Most companies do not need Snowflake or Databricks

#115

Post by ouraf »

That's common practice with mid sized companies and above, though:

- find something that can probably give some competitive advantage or please shareholders - Identify it as "not part of our core business" - Pay a third party company to do it even if it would be cheaper to things in house. If something goes wrong and the shareholders ask questions, the CEO can blame them instead of their own IT department and reassure them they're "focusing on what generates most value for the company".

Re: Most companies do not need Snowflake or Databricks

#116

Post by ledauphin »

jboggan wrote: post #52

Modifying the argument, I'd say even the companies that do need Snowflake or Databricks for an analytical use case tend to get run over by the hype train post adoption and start using it everywhere as an anti-pattern and causing much bigger problems operationally.

100%.

as an organization, there's value in _limiting_ the number of tools you adopt. But you'll nearly always want to adopt a few right-sized tools rather than using the same sledgehammer for every single thing that looks remotely like a nail.

Re: Most companies do not need Snowflake or Databricks

#117

Post by epups »

ramesh31 wrote: post #65
lr4444lr wrote: post #30

...a talk with your local Dev Ops Engineer/ manager to discuss how to secure your implementations This right here is exactly why Snowflake is a good fit for my org.: we could pay someone's salary to install, maintain, and upgrade some open source alternative (and the VPS to run it), or we could just pay for Snowflake and stop wasting engineering's time on the Data Science's team's stuff, which frees them up to move f…

>This right here is exactly why Snowflake is a good fit for my org.: we could pay someone's salary to install, maintain, and upgrade some open source alternative (and the VPS to run it), or we could just pay for Snowflake and stop wasting engineering's time on the Data Science's team's stuff, which frees them up to move faster on the core platform features bringing in the big bucks. The cost problem really is out of…

> DBUs are on the order of ~100x the actual underlying compute cost

Where did you get that from? Sounds like it can only be true if you only take into account electricity costs, not capital or wages.

Re: Most companies do not need Snowflake or Databricks

#118

Post by sanderjd »

krasznahorkai wrote: post #19

I work as a solution architect at a consulting firm that builds analytical data platforms for customers. Our company has a partnership with Snowflake, which means all the solutions we build are pushed to use Snowflake. Their sales strategy is very Oracle-like and at least in my circles many Snowflake sales employees are ex-Oracle. This means our sales and Snowflake sales are the best of friends. Formally they'll deny…

> you pay relatively little if you do relatively little data processing

Yep, surprised how few people have been mentioning this here.

Compared with running a managed postgres instance 100% of the time, running a snowflake xs warehouse for a few minutes a day can be significantly cheaper.

Re: Most companies do not need Snowflake or Databricks

#119

Post by mmcnl »

catchnear4321 wrote: post #32

> The cost for something that can be replicated free and open source is absurd. open source it may be. free it is not. paying an expert to correctly deploy an open source solution takes time and money. oh you want it maintained? the three recommendations sound like those of a consultant. they work great with exec buy-in, and are a joke without. yes, yes, you just have to explain yourself. just help management underst…

Big corps are risk management companies. The happy path is that you will have a working solution at a fraction of the cost. Mid to worst case scenario is that you end up over your head, the costs are higher than you anticipated and the end result is mediocre at best. Big corps are willing to pay a high premium to guarantuee a certain level of performance with penalties if necessary.

Re: Most companies do not need Snowflake or Databricks

#120

Post by mmcnl »

catchnear4321 wrote: post #32

> The cost for something that can be replicated free and open source is absurd. open source it may be. free it is not. paying an expert to correctly deploy an open source solution takes time and money. oh you want it maintained? the three recommendations sound like those of a consultant. they work great with exec buy-in, and are a joke without. yes, yes, you just have to explain yourself. just help management underst…

Big corps are risk management companies. The happy path is that you will have a working solution at a fraction of the cost. Mid to worst case scenario is that you end up over your head, the costs are higher than you anticipated and the end result is mediocre at best. Big corps are willing to pay a high premium to guarantuee a certain level of performance with penalties if necessary, especially if the solution is pretty far removed from their core business.
Post reply on HN