Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Most companies do not need Snowflake or Databricks

kjhealey.medium.com

Post reply on HN
Print view View original

81–90 of 140 posts

Re: Most companies do not need Snowflake or Databricks

#81

Post by commandlinefan »

catchnear4321 wrote: post #32

> The cost for something that can be replicated free and open source is absurd. open source it may be. free it is not. paying an expert to correctly deploy an open source solution takes time and money. oh you want it maintained? the three recommendations sound like those of a consultant. they work great with exec buy-in, and are a joke without. yes, yes, you just have to explain yourself. just help management underst…

> paying an expert to correctly deploy

That's what the Databricks salespeople say. My consistent experience has been that the experts "learned" it over the weekend by reading over the brochure and now I not only need to get a deep understanding of it myself I also have to waste my time explaining it to them because they have to actually do it in the closed environment the Databricks is overcharging us for.

Re: Most companies do not need Snowflake or Databricks

#82

Post by doctorpangloss »

catchnear4321 wrote: post #32

> The cost for something that can be replicated free and open source is absurd. open source it may be. free it is not. paying an expert to correctly deploy an open source solution takes time and money. oh you want it maintained? the three recommendations sound like those of a consultant. they work great with exec buy-in, and are a joke without. yes, yes, you just have to explain yourself. just help management underst…

I don't need a degree in Individual Contributorism with a citation in Consultantology to know that Snowflake and Databricks are ripping people off.

I'm not sure why anyone would be especially passionate about figuring out the details how though. If you want to level a criticism at the author, it's spending time on blog posts instead of ostensibly making the super cheap, better data analytics platform he's advocating for.

Re: Most companies do not need Snowflake or Databricks

#83

Post by bdcravens »

commandlinefan wrote: post #81
catchnear4321 wrote: post #32

> The cost for something that can be replicated free and open source is absurd. open source it may be. free it is not. paying an expert to correctly deploy an open source solution takes time and money. oh you want it maintained? the three recommendations sound like those of a consultant. they work great with exec buy-in, and are a joke without. yes, yes, you just have to explain yourself. just help management underst…

> paying an expert to correctly deploy That's what the Databricks salespeople say. My consistent experience has been that the experts "learned" it over the weekend by reading over the brochure and now I not only need to get a deep understanding of it myself I also have to waste my time explaining it to them because they have to actually do it in the closed environment the Databricks is overcharging us for.

Sounds like you're now a Databricks expert. Time to go get that $300/hour consulting fee.

Re: Most companies do not need Snowflake or Databricks

#84

Post by onlyrealcuzzo »

commandlinefan wrote: post #80

> charging their customers for Ferraris I've seen this sort of analogy before and I don't think it's the right one here: if I get a Ferrari, I know _exactly_ what I'm getting. I'm getting a really fast, really beautiful car. I'm not too familiar with Snowflake, but I've suffered under Databricks on and off for about a decade now, and as far as I can tell, it's just a more expensive, closed way to do what I could do a…

Theoretically, your employer is paying Databricks, because they'd rather pay them than have a harder time finding employees with actual skills (and paying them accordingly).

Additionally, it's somewhat of a standard now - so it's easy to hire other people in the org with skills using it.

Re: Most companies do not need Snowflake or Databricks

#85

Post by [deleted] »

MrPowers wrote: post #70

I think most companies don't understand a large part of the Databricks offering and it should be used by way more organizations. Disclaimer: I was a Databricks user for 6 years and now work at Databricks. Yea, you can create your own Spark deployment, but it will run much slower than the Databricks Runtime (DBR) or the Databricks proprietary Spark Runtime (Photon). Computations that run slower cause you to have a lar…

[deleted]

Re: Most companies do not need Snowflake or Databricks

#86

Post by bob1029 »

Why not give in and pay for the SQL Server Hyperscale instance? Isn't there enough BS to worry about? Why continue to waste time on tired old OLTP/OLAP/scalability/etc. conversations in 2023?

Unless your business can continuously write >100 megabytes of transactional data per second, this solution would almost certainly address all of your needs forever and ever. Up to 100TB too. It just works. It offers transactions exactly the way most business expect them to be conducted. No weird code, no weird client libraries, nothing. It works more or less like it has for the last 30+ years.

I can tell you for a fact that simply setting up a gigantic shared database called "company" and getting the team connected to it did wonders for us. When you stop worrying about "will it scale" you can start to collaborate and do amazing things again.

Re: Most companies do not need Snowflake or Databricks

#87

Post by nightski »

catchnear4321 wrote: post #76
nightski wrote: post #67

Earlier quoted context omitted.

I think you are vastly under-estimating the amount of cost/manpower needed for snowflake and cloud based solutions (they are anything but turn-key and have a lot of churn). There are cases for both but it's not as simple as open source = hire and cloud = no people needed. With Snowflake you are paying for not only the development of the product, the hosting of it, the engineers to run it, but also the sales, marketin…

always fun when folks make assumptions. > With Snowflake you are paying for not only the development of the product, the hosting of it, the engineers to run it, but also the sales, marketing, and management behind it. and you can run the numbers to determine if the cost of ownership offsets headcount. what your dollar goes to isn’t part of that formula. only what your dollar gets you. > But on top of all that you pro…

We aren't talking about a homebrew data warehouse though, we are talking about open source software. Designing the schema and/or data flows on top of the warehouse is needed regardless of whether you use open source or Snowflake.

Re: Most companies do not need Snowflake or Databricks

#88

Post by catchnear4321 »

commandlinefan wrote: post #81
catchnear4321 wrote: post #32

> The cost for something that can be replicated free and open source is absurd. open source it may be. free it is not. paying an expert to correctly deploy an open source solution takes time and money. oh you want it maintained? the three recommendations sound like those of a consultant. they work great with exec buy-in, and are a joke without. yes, yes, you just have to explain yourself. just help management underst…

> paying an expert to correctly deploy That's what the Databricks salespeople say. My consistent experience has been that the experts "learned" it over the weekend by reading over the brochure and now I not only need to get a deep understanding of it myself I also have to waste my time explaining it to them because they have to actually do it in the closed environment the Databricks is overcharging us for.

was referring to the open source alternatives.

have yet to meet a vendor-recommended solutions provider that felt like more than a cash grab for anyone too dazzled by the initial onboarding.

the math of outsourcing implementation to consulting firms always seems off - send the problem elsewhere, usually for a premium, and hope the internal folks can digest whatever the actual consulting dev writes after the sales dev talks big game on hypotheticals.

hesitant to say my sample size is too small. feels like a trap.

Re: Most companies do not need Snowflake or Databricks

#89

Post by orochimaaru »

onlyrealcuzzo wrote: post #84
commandlinefan wrote: post #80

> charging their customers for Ferraris I've seen this sort of analogy before and I don't think it's the right one here: if I get a Ferrari, I know _exactly_ what I'm getting. I'm getting a really fast, really beautiful car. I'm not too familiar with Snowflake, but I've suffered under Databricks on and off for about a decade now, and as far as I can tell, it's just a more expensive, closed way to do what I could do a…

Theoretically, your employer is paying Databricks, because they'd rather pay them than have a harder time finding employees with actual skills (and paying them accordingly). Additionally, it's somewhat of a standard now - so it's easy to hire other people in the org with skills using it.

But isn’t that something they would need to do anyway - find employees with actual skills?

I mean sure, it’s how the snake oil is sold - “buy so and so saas product and you wouldn’t need to hire and can reduce cost”. I’m surprised that line can still sell today because in software it’s rarely true unless you can merge excellent talent with said product.

Re: Most companies do not need Snowflake or Databricks

#90

Post by onlyrealcuzzo »

orochimaaru wrote: post #89
onlyrealcuzzo wrote: post #84

Earlier quoted context omitted.

Theoretically, your employer is paying Databricks, because they'd rather pay them than have a harder time finding employees with actual skills (and paying them accordingly). Additionally, it's somewhat of a standard now - so it's easy to hire other people in the org with skills using it.

But isn’t that something they would need to do anyway - find employees with actual skills? I mean sure, it’s how the snake oil is sold - “buy so and so saas product and you wouldn’t need to hire and can reduce cost”. I’m surprised that line can still sell today because in software it’s rarely true unless you can merge excellent talent with said product.

If you work at a small startup - they are mostly parroting what large and mid-size companies are doing - regardless of the fact that they're likely to fail before they get to that size, and doing said thing might make their chances of failure higher.

If you work at a large company - having a larger pool of candidates to hire from is definitely worthwhile.

And, it is much easier to have a poor understanding of Databricks than a very good understanding of data engineering in general.

Someone with a poor understanding of Databricks might be able to do 80% of what you can do with a great understanding of data engineering in general.

Post reply on HN