Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Most companies do not need Snowflake or Databricks

kjhealey.medium.com

Post reply on HN
Print view View original

51–60 of 140 posts

Re: Most companies do not need Snowflake or Databricks

#51

Post by marginalia_nu »

marginalia_nu wrote: post #25
nottorp wrote: post #11

99.99% of companies are not Google sized and don't need Google's solutions. Period.

Google-sized solutions tend to create the need for google sized solutions though. Take a basic CRUD TODO app. Add metrics to every user interaction, every mouse movement. Now you need a time series database, kafka and a data lake. Break it down into microservices. Now you need kubernetes, and also structured logging, and probably an ELK stack. You also probably need Graphana to monitor your services. Because you have…

Thinking about it, this may be a corollary of Parkinson's law.

Re: Most companies do not need Snowflake or Databricks

#53

Post by eitally »

benrutter wrote: post #49

As a data engineer who's worked in a bunch of different contexts/companies I 100% agree most of the time snowflake/databricks is an unnecessary money sink. The main problem is most companies need the security of a managed service for cloud computing, and don't want to be locked out of scaling to a very large scale (with distributed compute). Unfortunately, I don't think there are a lot of options that meet those pret…

I think your second point is vastly overstated for 99% of companies and their use cases. There are precious few situations that can't be handled decently well by a simple RDBMS, and a simple ETL process into an analytical data store for reporting. Almost everything that exist fits one of three buckets: 1) CRUD app, 2) reporting tool, 3) business process app. And almost all of this is for internal use only. It's really only a handful of enterprise things that need massive scalability and performance, and consumer-oriented stuff (much of which is pretty niche and also doesn't need massive global scalability).

Re: Most companies do not need Snowflake or Databricks

#54

Post by catchnear4321 »

pydry wrote: post #47
catchnear4321 wrote: post #32

> The cost for something that can be replicated free and open source is absurd. open source it may be. free it is not. paying an expert to correctly deploy an open source solution takes time and money. oh you want it maintained? the three recommendations sound like those of a consultant. they work great with exec buy-in, and are a joke without. yes, yes, you just have to explain yourself. just help management underst…

>open source it may be. free it is not. paying an expert to correctly deploy an open source solution takes time and money. oh you want it maintained? Funny. I heard this same kind of argument used against replacing oracle with postgres. It reminds me of Microsoft's "TCO" PR offensive back when they were public about how much they loathed open source competition. Thing is that it wasnt just a straw man (nobody that ne…

context matters.

you’re calling it an organization, that alone typically indicates a larger scale, in which supporting something internal might be feasible. or at least it is a term used by those that have been there. you’re also referencing politics, which, again, that’s highly suggestive of a specific type of experience. experience within scale if not at scale.

please don’t mistake my intent to be that open source never makes sense. with the right plan and personnel, it can work better.

with the right scale and support, open source projects are started, though they aren’t always open from day one.

Re: Most companies do not need Snowflake or Databricks

#55

Post by eitally »

marginalia_nu wrote: post #51
marginalia_nu wrote: post #25

Earlier quoted context omitted.

Google-sized solutions tend to create the need for google sized solutions though. Take a basic CRUD TODO app. Add metrics to every user interaction, every mouse movement. Now you need a time series database, kafka and a data lake. Break it down into microservices. Now you need kubernetes, and also structured logging, and probably an ELK stack. You also probably need Graphana to monitor your services. Because you have…

Thinking about it, this may be a corollary of Parkinson's law.

It definitely is. It also sort of applies to legacy enterprise, too, in a different flavor: "If the tool or system is easy to use and accessible (thing classic LAMP stack), then shadow IT is guaranteed to exist and start using it." The more modern version flavor is that "if a cloud service exists, then your enterprise architecture will expand to consume it [whether it's necessary or not]."

Re: Most companies do not need Snowflake or Databricks

#56

Post by catchnear4321 »

code51 wrote: post #44
catchnear4321 wrote: post #37

Earlier quoted context omitted.

years ago, had this one college professor, students would spend the time before class asking him to divide two fractional numbers, and he would rattle off a few decimal places after calculating, to cheers from the students that had their calculators out with the answers. the excitement would build with every correct decimal point. he didn’t actually need a calculator. but he did own one. hopefully this does not cause…

Am I right in understanding that all these companies are actually making deals for on-demand use in the future?

you can get started with many of these companies with just a credit card with monthly billing. just like the major cloud vendors.

Re: Most companies do not need Snowflake or Databricks

#57

Post by PurpleRamen »

mrweasel wrote: post #34

Most companies don't need a lot of things. I don't do a whole lot of data analytics anymore, but I'd say: Start with figuring out how much data you actually have. We still see companies claim to have vast amounts of data, but in reality they are talking about less than a TB of data, frequently just a few 100GB. When you operate at that scale, just chuck your data into whatever database you're comfortable with and do…

A Terabyte of data can be still vast, depending on how different and unstructured they are. If you have thousands of different logfiles-formats, datadump-formats and various complex fileformats, then you need a very different solution than you would need to handle a terabyte of uniform data in a csv-file or similar.

Re: Most companies do not need Snowflake or Databricks

#58

Post by peteradio »

Aurornis wrote: post #43
mrweasel wrote: post #34

Most companies don't need a lot of things. I don't do a whole lot of data analytics anymore, but I'd say: Start with figuring out how much data you actually have. We still see companies claim to have vast amounts of data, but in reality they are talking about less than a TB of data, frequently just a few 100GB. When you operate at that scale, just chuck your data into whatever database you're comfortable with and do…

> We still see companies claim to have vast amounts of data, but in reality they are talking about less than a TB of data, frequently just a few 100GB. When you operate at that scale, just chuck your data into whatever database you're comfortable with and do SQL queries, it's fine. I did some work with a company that talked about their “big data”, bragged about their data lake, and made grandiose statements about how…

Remember some industries are perversely incentivized to drive up costs because profits are limited as a percentage of total cost.

Re: Most companies do not need Snowflake or Databricks

#59

Post by sseagull »

mrweasel wrote: post #34

Most companies don't need a lot of things. I don't do a whole lot of data analytics anymore, but I'd say: Start with figuring out how much data you actually have. We still see companies claim to have vast amounts of data, but in reality they are talking about less than a TB of data, frequently just a few 100GB. When you operate at that scale, just chuck your data into whatever database you're comfortable with and do…

I find it frustrating to search for stuff for mid-scale problems sometimes. There's a lack of context about what 'metrics' or 'analytics' means, and that leads to over-engineered solutions.

I run 2-3 (primarily database & web) servers. I thought it would be a nice idea to aggregate web accesses and stuff into a central location for analysis. It's probably 50-100MB of logs a day at most. But all the 'industry standard' solutions are for something at much bigger scale. Start with blah, spin that up on a k8s cluster, run this in docker, blah blah. FFS, I don't need that, I just want some nice graphs.

So I just shove it all into a postgres database (since we are already familiar with it) and visualize it with Grafana. I'm sure people would complain that it's not the proper way, but I prefer spending my time solving my users problems, not self-inflicted ones.

Re: Most companies do not need Snowflake or Databricks

#60

Post by rglullis »

lr4444lr wrote: post #30

...a talk with your local Dev Ops Engineer/ manager to discuss how to secure your implementations This right here is exactly why Snowflake is a good fit for my org.: we could pay someone's salary to install, maintain, and upgrade some open source alternative (and the VPS to run it), or we could just pay for Snowflake and stop wasting engineering's time on the Data Science's team's stuff, which frees them up to move f…

> we could pay someone's salary to install, maintain, and upgrade some open source alternative (and the VPS to run it), or we could just pay for Snowflake and stop wasting engineering's time

You are missing the "we could pay someone that provides a commodified, open-source alternative to Snowflake" option.

It would be cheaper, it wouldn't require any of your engineering team resources and it would help us all get rid of single vendor lock-in.

Post reply on HN