Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Most companies do not need Snowflake or Databricks

kjhealey.medium.com

Post reply on HN
Print view View original

121–130 of 140 posts

Re: Most companies do not need Snowflake or Databricks

#121

Post by riku_iki »

mmcnl wrote: post #120
catchnear4321 wrote: post #32

> The cost for something that can be replicated free and open source is absurd. open source it may be. free it is not. paying an expert to correctly deploy an open source solution takes time and money. oh you want it maintained? the three recommendations sound like those of a consultant. they work great with exec buy-in, and are a joke without. yes, yes, you just have to explain yourself. just help management underst…

Big corps are risk management companies. The happy path is that you will have a working solution at a fraction of the cost. Mid to worst case scenario is that you end up over your head, the costs are higher than you anticipated and the end result is mediocre at best. Big corps are willing to pay a high premium to guarantuee a certain level of performance with penalties if necessary, especially if the solution is pret…

So, how these risk managers jump on new hyped tech, built by non profitable companies with vendor lock in and no way to escape?

Re: Most companies do not need Snowflake or Databricks

#122

Post by dangjc »

ramesh31 wrote: post #65
lr4444lr wrote: post #30

...a talk with your local Dev Ops Engineer/ manager to discuss how to secure your implementations This right here is exactly why Snowflake is a good fit for my org.: we could pay someone's salary to install, maintain, and upgrade some open source alternative (and the VPS to run it), or we could just pay for Snowflake and stop wasting engineering's time on the Data Science's team's stuff, which frees them up to move f…

>This right here is exactly why Snowflake is a good fit for my org.: we could pay someone's salary to install, maintain, and upgrade some open source alternative (and the VPS to run it), or we could just pay for Snowflake and stop wasting engineering's time on the Data Science's team's stuff, which frees them up to move faster on the core platform features bringing in the big bucks. The cost problem really is out of…

DBUs are about +50% the underlying compute cost.

We use Databricks in production extensively.

Re: Most companies do not need Snowflake or Databricks

#123

Post by agent281 »

sanderjd wrote: post #102
agent281 wrote: post #78

Earlier quoted context omitted.

If you have petabytes of data, I don't think this article is talking about your use case.

I think it is? Or I guess, what data size do you think it's talking about? If you only have gigabytes of data, none of this matters, you can use anything pretty cheaply and easily. So is this article just for "terabytes" or does it go up to "hundreds of terabytes" but not "petabytes"?

Hmm, I suppose it's a bit challenging to say. I initially thought that it wasn't for the 80% smallest companies and petabytes of data is probably puts you in the top 20%. (Most businesses are small businesses after all.)

However, I now realize that th biggest companies probably should manage their own data. If you're Google why would you use Snowflake?

So I don't know if you are the target audience for this blog post. It's pretty ambiguous.

Re: Most companies do not need Snowflake or Databricks

#124

Post by chkelly9 »

catchnear4321 wrote: post #32

> The cost for something that can be replicated free and open source is absurd. open source it may be. free it is not. paying an expert to correctly deploy an open source solution takes time and money. oh you want it maintained? the three recommendations sound like those of a consultant. they work great with exec buy-in, and are a joke without. yes, yes, you just have to explain yourself. just help management underst…

[Disclaimer: I work at ClickHouse] I see a lot of the responses here focus on the age-old ‘build v. buy’ debate.

It’s also worth considering the comparative cost of Snowflake against other saas warehouses or databases, depending on your needs. For instance, we’ve heard from users that ClickHouse Cloud can be much more cost-effective for many use cases when compared to Snowflake - real-time analytics is a great example. For those who can’t build (or run OSS themselves), this is another interesting (and important) dimension. What's great about ClickHouse is that it's open source, too :).

Re: Most companies do not need Snowflake or Databricks

#125

Post by notatoad »

rglullis wrote: post #60
lr4444lr wrote: post #30

...a talk with your local Dev Ops Engineer/ manager to discuss how to secure your implementations This right here is exactly why Snowflake is a good fit for my org.: we could pay someone's salary to install, maintain, and upgrade some open source alternative (and the VPS to run it), or we could just pay for Snowflake and stop wasting engineering's time on the Data Science's team's stuff, which frees them up to move f…

> we could pay someone's salary to install, maintain, and upgrade some open source alternative (and the VPS to run it), or we could just pay for Snowflake and stop wasting engineering's time You are missing the "we could pay someone that provides a commodified, open-source alternative to Snowflake" option. It would be cheaper, it wouldn't require any of your engineering team resources and it would help us all get rid…

>it wouldn't require any of your engineering team resources

other than the resources to evaluate the commodified open-source alternative. and the resources to manage the risk of that alternative, and the smaller company who runs it, going away.

"nobody ever got fired for buying IBM" is usually said in a snarky way, but there's real value in not only not having to think about the provider, but not having to think about which provider to choose. making that decision has a cost, and the more off the beaten path you go the higher the cost of that decision.

Re: Most companies do not need Snowflake or Databricks

#126

Post by hodgesrm »

sanderjd wrote: post #118
krasznahorkai wrote: post #19

I work as a solution architect at a consulting firm that builds analytical data platforms for customers. Our company has a partnership with Snowflake, which means all the solutions we build are pushed to use Snowflake. Their sales strategy is very Oracle-like and at least in my circles many Snowflake sales employees are ex-Oracle. This means our sales and Snowflake sales are the best of friends. Formally they'll deny…

> you pay relatively little if you do relatively little data processing Yep, surprised how few people have been mentioning this here. Compared with running a managed postgres instance 100% of the time, running a snowflake xs warehouse for a few minutes a day can be significantly cheaper.

The reason is that Snowflake charges close to cost for object storage and they compress your data when they put it in.

However...that means they need to make it up elsewhere. When you do get around to running queries the markup on compute can be 5-10x (or potentially more) depending on the plan you are using. If you do constant, compute heavy aggregation Snowflake is not the right place to do it.

Re: Most companies do not need Snowflake or Databricks

#127

Post by buttaphingas »

chkelly9 wrote: post #124
catchnear4321 wrote: post #32

> The cost for something that can be replicated free and open source is absurd. open source it may be. free it is not. paying an expert to correctly deploy an open source solution takes time and money. oh you want it maintained? the three recommendations sound like those of a consultant. they work great with exec buy-in, and are a joke without. yes, yes, you just have to explain yourself. just help management underst…

[Disclaimer: I work at ClickHouse] I see a lot of the responses here focus on the age-old ‘build v. buy’ debate. It’s also worth considering the comparative cost of Snowflake against other saas warehouses or databases, depending on your needs. For instance, we’ve heard from users that ClickHouse Cloud can be much more cost-effective for many use cases when compared to Snowflake - real-time analytics is a great exampl…

Vendor says their solution is cheaper than competitor solution. I'm shocked.

Re: Most companies do not need Snowflake or Databricks

#128

Post by rglullis »

notatoad wrote: post #125
rglullis wrote: post #60

Earlier quoted context omitted.

> we could pay someone's salary to install, maintain, and upgrade some open source alternative (and the VPS to run it), or we could just pay for Snowflake and stop wasting engineering's time You are missing the "we could pay someone that provides a commodified, open-source alternative to Snowflake" option. It would be cheaper, it wouldn't require any of your engineering team resources and it would help us all get rid…

>it wouldn't require any of your engineering team resources other than the resources to evaluate the commodified open-source alternative. and the resources to manage the risk of that alternative, and the smaller company who runs it, going away. "nobody ever got fired for buying IBM" is usually said in a snarky way, but there's real value in not only not having to think about the provider, but not having to think abou…

> "nobody ever got fired for buying IBM"

https://en.wikipedia.org/wiki/File:Survivorship-bias.svg

How many startups simply died or how many projects never got to be launched because they thought they needed some enterprise support from the start?

Re: Most companies do not need Snowflake or Databricks

#129

Post by sanderjd »

agent281 wrote: post #123
sanderjd wrote: post #102

Earlier quoted context omitted.

I think it is? Or I guess, what data size do you think it's talking about? If you only have gigabytes of data, none of this matters, you can use anything pretty cheaply and easily. So is this article just for "terabytes" or does it go up to "hundreds of terabytes" but not "petabytes"?

Hmm, I suppose it's a bit challenging to say. I initially thought that it wasn't for the 80% smallest companies and petabytes of data is probably puts you in the top 20%. (Most businesses are small businesses after all.) However, I now realize that th biggest companies probably should manage their own data. If you're Google why would you use Snowflake? So I don't know if you are the target audience for this blog post…

I guess I'll say what I think. I do think it is targeted at that smallest 80% of companies with some digital footprint, and also at most of the top 20%. Or more specifically, I think maybe it's targeted at like the 5th percentile to the 99th percentile. That bottom 5% probably just needs Excel, and that top 1% is probably writing or heavily modifying all their own tools.

But I'm not sure the advice is very good from the 5th percentile up to ... maybe that top 20%? A lot of the stuff in the article assumes the availability of sophisticated data architects and mature infrastructure groups that I really don't think the median company has.

Re: Most companies do not need Snowflake or Databricks

#130

Post by agent281 »

sanderjd wrote: post #129
agent281 wrote: post #123

Earlier quoted context omitted.

Hmm, I suppose it's a bit challenging to say. I initially thought that it wasn't for the 80% smallest companies and petabytes of data is probably puts you in the top 20%. (Most businesses are small businesses after all.) However, I now realize that th biggest companies probably should manage their own data. If you're Google why would you use Snowflake? So I don't know if you are the target audience for this blog post…

I guess I'll say what I think. I do think it is targeted at that smallest 80% of companies with some digital footprint, and also at most of the top 20%. Or more specifically, I think maybe it's targeted at like the 5th percentile to the 99th percentile. That bottom 5% probably just needs Excel, and that top 1% is probably writing or heavily modifying all their own tools. But I'm not sure the advice is very good from…

I agree. Really seasoned data people are not common enough. Small companies need to buy services to lighten the load.

We both seem have a sense of the size of companies at different percentiles. At what percentile would you put your company with petabytes of data?

Post reply on HN