Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Most companies do not need Snowflake or Databricks

kjhealey.medium.com

Post reply on HN
Print view View original

101–110 of 140 posts

Re: Most companies do not need Snowflake or Databricks

#101

Post by el_oni »

MrPowers wrote: post #70

I think most companies don't understand a large part of the Databricks offering and it should be used by way more organizations. Disclaimer: I was a Databricks user for 6 years and now work at Databricks. Yea, you can create your own Spark deployment, but it will run much slower than the Databricks Runtime (DBR) or the Databricks proprietary Spark Runtime (Photon). Computations that run slower cause you to have a lar…

I begrudgingly agree. We tried to stand up a spark cluster and there are just loads of configuration and little niggles that just get in the way, like making sure everything on your cluster has the libraries they need.

Putting it on databricks most of those problems went away and we could just get in with writing the code.

Could we have figured out how to get spark working in a bespoke configuration? almost certainly. By our mid September deadline? Probably not

Re: Most companies do not need Snowflake or Databricks

#102

Post by sanderjd »

agent281 wrote: post #78
ironchef wrote: post #69

Earlier quoted context omitted.

Here was my situation. Occasional queries. Over a couple petabyte of data. Customer facing so response in seconds per SLA but > 95 percent of the time the warehouse isn’t running. Cached queries from within 24 hours which don’t require the warehouse to even spin up. Our snowflake costs were significantly less than an FTE. Would that potentially be a situation which “running your own” doesn’t make sense?

If you have petabytes of data, I don't think this article is talking about your use case.

I think it is?

Or I guess, what data size do you think it's talking about? If you only have gigabytes of data, none of this matters, you can use anything pretty cheaply and easily. So is this article just for "terabytes" or does it go up to "hundreds of terabytes" but not "petabytes"?

Re: Most companies do not need Snowflake or Databricks

#103

Post by kristjansson »

ramesh31 wrote: post #65
lr4444lr wrote: post #30

...a talk with your local Dev Ops Engineer/ manager to discuss how to secure your implementations This right here is exactly why Snowflake is a good fit for my org.: we could pay someone's salary to install, maintain, and upgrade some open source alternative (and the VPS to run it), or we could just pay for Snowflake and stop wasting engineering's time on the Data Science's team's stuff, which frees them up to move f…

>This right here is exactly why Snowflake is a good fit for my org.: we could pay someone's salary to install, maintain, and upgrade some open source alternative (and the VPS to run it), or we could just pay for Snowflake and stop wasting engineering's time on the Data Science's team's stuff, which frees them up to move faster on the core platform features bringing in the big bucks. The cost problem really is out of…

Where do you see DBUs being many-times multiples of the underlying compute? AFAICT they're basically a x% (for x Of course, there's the argument that AWS et al are overpriced relative to running your own hardware... which this community can argue ad infinitum, but seems besides the point here.

Re: Most companies do not need Snowflake or Databricks

#104

Post by sanderjd »

sseagull wrote: post #59
mrweasel wrote: post #34

Most companies don't need a lot of things. I don't do a whole lot of data analytics anymore, but I'd say: Start with figuring out how much data you actually have. We still see companies claim to have vast amounts of data, but in reality they are talking about less than a TB of data, frequently just a few 100GB. When you operate at that scale, just chuck your data into whatever database you're comfortable with and do…

I find it frustrating to search for stuff for mid-scale problems sometimes. There's a lack of context about what 'metrics' or 'analytics' means, and that leads to over-engineered solutions. I run 2-3 (primarily database & web) servers. I thought it would be a nice idea to aggregate web accesses and stuff into a central location for analysis. It's probably 50-100MB of logs a day at most. But all the 'industry standard…

I feel this in my bones. I really find it hard to figure out the right solutions for scaling things that will be "bigger than small" but also "smaller than big" for the foreseeable future. That is, when the "starting point" conventional wisdom ("just use postgres / sqlite for everything; anything else is yagni") isn't quite up to snuff, but the "scale up" solutions are designed to handle far bigger use cases and bring some baggage along with that.

I have honestly found Snowflake pretty useful for this, because it seems pretty cheap to me for a small amount of usage, but is also elastic up to this middle-ground of usage. But I suspect that at "truly big" I would / will be interested in different solutions.

Re: Most companies do not need Snowflake or Databricks

#105

Post by mrweasel »

sseagull wrote: post #59
mrweasel wrote: post #34

Most companies don't need a lot of things. I don't do a whole lot of data analytics anymore, but I'd say: Start with figuring out how much data you actually have. We still see companies claim to have vast amounts of data, but in reality they are talking about less than a TB of data, frequently just a few 100GB. When you operate at that scale, just chuck your data into whatever database you're comfortable with and do…

I find it frustrating to search for stuff for mid-scale problems sometimes. There's a lack of context about what 'metrics' or 'analytics' means, and that leads to over-engineered solutions. I run 2-3 (primarily database & web) servers. I thought it would be a nice idea to aggregate web accesses and stuff into a central location for analysis. It's probably 50-100MB of logs a day at most. But all the 'industry standard…

For the graphs I don't know, but I was happy to learn that Journald can actually ship logs to other Journald installations. Haven't used it yet, but I'm still looking of a place where it would make sense.

Re: Most companies do not need Snowflake or Databricks

#106

Post by rglullis »

bushbaba wrote: post #99
rglullis wrote: post #60

Earlier quoted context omitted.

> we could pay someone's salary to install, maintain, and upgrade some open source alternative (and the VPS to run it), or we could just pay for Snowflake and stop wasting engineering's time You are missing the "we could pay someone that provides a commodified, open-source alternative to Snowflake" option. It would be cheaper, it wouldn't require any of your engineering team resources and it would help us all get rid…

Would it be cheaper? Cloudera fits the bill and it’s for sure not cheaper

Because Cloudera is aimed at a market that is not the same that would benefit from "most companies" being talked about in TFA?

Plenty of companies that would be fine by using a managed postgresql database for their "data warehouse" and you can get, e.g, one reasonably powerful server at Digital Ocean for $60/month with managed backups - and that already has quite a big markup compared to their droplets because they are just packaging some monitoring tools on top of their existing systems.

Re: Most companies do not need Snowflake or Databricks

#107

Post by hodgesrm »

rglullis wrote: post #60
lr4444lr wrote: post #30

...a talk with your local Dev Ops Engineer/ manager to discuss how to secure your implementations This right here is exactly why Snowflake is a good fit for my org.: we could pay someone's salary to install, maintain, and upgrade some open source alternative (and the VPS to run it), or we could just pay for Snowflake and stop wasting engineering's time on the Data Science's team's stuff, which frees them up to move f…

> we could pay someone's salary to install, maintain, and upgrade some open source alternative (and the VPS to run it), or we could just pay for Snowflake and stop wasting engineering's time You are missing the "we could pay someone that provides a commodified, open-source alternative to Snowflake" option. It would be cheaper, it wouldn't require any of your engineering team resources and it would help us all get rid…

> You are missing the "we could pay someone that provides a commodified, open-source alternative to Snowflake" option.

Which services provider do you have in mind to pay? The advantage of Snowflake is that the CIO can pick Snowflake as the default for the entire org and it will work across a very large number of business use cases. I don't see any open source analytic software that can do that.

In my experience open source analytic solutions work much better for specific use cases, say network flow log analysis. You pick exactly the right set of components for that use case and build specifically to the problem. Maybe you use ClickHouse + Kafka--cheaper, simpler, faster. But when you shift to another problem, such as analyzing support case patterns, that requires unpredictable joins on large numbers of tables the first solution does not work at all. Now you need something like Presto on data lakes, which is completely different technology.

The alternative is to move all the data in to Snowflake and do both of the above use cases there. Sure, it's not the fastest or the cheapest software but the savings in labor more than make up for the licensing cost. A lot of companies make that decision.

Re: Most companies do not need Snowflake or Databricks

#108

Post by hodgesrm »

commandlinefan wrote: post #80

> charging their customers for Ferraris I've seen this sort of analogy before and I don't think it's the right one here: if I get a Ferrari, I know _exactly_ what I'm getting. I'm getting a really fast, really beautiful car. I'm not too familiar with Snowflake, but I've suffered under Databricks on and off for about a decade now, and as far as I can tell, it's just a more expensive, closed way to do what I could do a…

The car example in the article is exactly backwards. Snowflake is the Toyota Camry. It's a mass market solution that fits a lot of basic use cases. Readers are conflating the licensing cost of Snowflake with total cost of ownership, which is quite different. The equation tilts further toward Snowflake as you load more data into it. It has a pretty efficient sharing model using virtual data warehouses.

If you need the Ferrari, that's when you build with open source. If I want real-time response at fixed cost on large amounts of data I'm going to take ClickHouse with a customized stack every time. But not every problem needs that solution.

(I work at a ClickHouse vendor and know its habits. But you could say the same about Druid or other databases.)

Re: Most companies do not need Snowflake or Databricks

#109

Post by polskibus »

kfk wrote: post #26
polskibus wrote: post #18

Earlier quoted context omitted.

I guess the ETL orchestration was the hard part? How did you do it?

Connectivity was the hardest part, we had to write Python connectors for a variety of ERP systems. We have a 1 server setup where we run duckdb and Python scripts, we monitor and orchestrate with Prefect but you can use one of the many Python orchestration tools. We load the finished data marts to a MS SQL server and users connect to it via PowerBI or Excel.

I don't understand why would you use duckdb as an intermediate step to fill Ms SQL DW. Surely you could just go to Ms SQL directly?

Re: Most companies do not need Snowflake or Databricks

#110

Post by its_a_random_ac »

HorizonXP wrote: post #46

My client is in the middle of migrating their Snowflake data to Databricks Unity Catalog. To this day, I still do not understand why they love Databricks so much. It just looks like a Jupyter notebook to me. I know, it has Spark. So? I’m trying to learn more so I can build something better, maybe.

Yeah, I'd like to know too, from everything I can see it's just a cloud-hosted notebook with a spark backend, which does have some benefits, but I can't tell why they got as big as they did.

Also from a more cynical side, I've interviewed at a lot of companies that ask if I've used it before, and I'm really temped to just lie, since it feels like something that could be picked up really easily.

Post reply on HN