Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Most companies do not need Snowflake or Databricks

kjhealey.medium.com

Post reply on HN
Print view View original

91–100 of 140 posts

Re: Most companies do not need Snowflake or Databricks

#91

Post by ebiester »

catchnear4321 wrote: post #32

> The cost for something that can be replicated free and open source is absurd. open source it may be. free it is not. paying an expert to correctly deploy an open source solution takes time and money. oh you want it maintained? the three recommendations sound like those of a consultant. they work great with exec buy-in, and are a joke without. yes, yes, you just have to explain yourself. just help management underst…

I think that's fair. I also think it's fair to count on-prem versus hosted as a metric for your org that should be evaluated.

I think that what is missing from the article is that it's not a matter of free versus paid, but rather integrated solution versus dispersed solution with custom glue code. Depending on your organization and your constraints, you should define your budget, wants and needs, build metrics to represent those, evaluate all systems against your particular metrics, and come to a dispassionate result.

Re: Most companies do not need Snowflake or Databricks

#92

Post by catchnear4321 »

nightski wrote: post #87
catchnear4321 wrote: post #76

Earlier quoted context omitted.

always fun when folks make assumptions. > With Snowflake you are paying for not only the development of the product, the hosting of it, the engineers to run it, but also the sales, marketing, and management behind it. and you can run the numbers to determine if the cost of ownership offsets headcount. what your dollar goes to isn’t part of that formula. only what your dollar gets you. > But on top of all that you pro…

We aren't talking about a homebrew data warehouse though, we are talking about open source software. Designing the schema and/or data flows on top of the warehouse is needed regardless of whether you use open source or Snowflake.

if it isn’t already running, yes, we are. open source is rarely if ever “off the shelf.”

if you’re suggesting vendor-hosted open source, that’s a consideration in vendor selection, and as much as it may seem obvious to suggest alternatives now, it wasn’t when they didn’t exist.

and yes, you correctly gathered the point on schemas and the like being the same everywhere.

Re: Most companies do not need Snowflake or Databricks

#94

Post by adam_oxla »

While I do agree that for most of companies those large SaaS solutions are overkill I do not think that DuckDB or similar is sufficient. Nowadays more of the companies really need to process large datasets.

I meet regularly companies that used PostgreSQL or something similar up to some point but then they have grown and it is not sufficient anymore. They need something scalable. It does not have to be large SaaS: in many cases small Clickhouse cluster is sufficient. Nevertheless not everything can be done using single server. Also even if customer knows exactly well what are their needs right now does needs will grow and change over time so it is reasonable to build something that is not only good enough for now. Of course building something absolutely "future proof" leads to extremes and high bills.

Re: Most companies do not need Snowflake or Databricks

#95

Post by slotrans »

Basically correct. If you have less than tens-of-terabytes the big exciting stuff just isn't needed and isn't particularly helpful.

Focus on understanding your data and building a useful model.

Use Postgres if you can. Supplement with DuckDB and/or ClickHouse as needed. Or, use your cloud's columnar DB (Redshift, BQ, Azure whatever) because you can start and stop using it as you please without talking to sales or signing a contract.

If your data team doesn't have the requisite skills... well... consider that tech selection might not be your problem.

Re: Most companies do not need Snowflake or Databricks

#96

Post by bick_nyers »

bob1029 wrote: post #86

Why not give in and pay for the SQL Server Hyperscale instance? Isn't there enough BS to worry about? Why continue to waste time on tired old OLTP/OLAP/scalability/etc. conversations in 2023? Unless your business can continuously write >100 megabytes of transactional data per second , this solution would almost certainly address all of your needs forever and ever. Up to 100TB too. It just works. It offers transaction…

If you do magically hit the "unicorn vector" and 100x your customer-base then you can start worrying about "will it scale", and thankfully you will have new revenue that can be deployed in the form of new hardware and new hires.

Re: Most companies do not need Snowflake or Databricks

#98

Post by bushbaba »

orochimaaru wrote: post #89
onlyrealcuzzo wrote: post #84

Earlier quoted context omitted.

Theoretically, your employer is paying Databricks, because they'd rather pay them than have a harder time finding employees with actual skills (and paying them accordingly). Additionally, it's somewhat of a standard now - so it's easy to hire other people in the org with skills using it.

But isn’t that something they would need to do anyway - find employees with actual skills? I mean sure, it’s how the snake oil is sold - “buy so and so saas product and you wouldn’t need to hire and can reduce cost”. I’m surprised that line can still sell today because in software it’s rarely true unless you can merge excellent talent with said product.

We outsource janitors, lunches, and other aspects of the business. Some things are not your core competency where having additional people investment detracts from your core mission.

That and do you really think 99% of companies are able to build something better or cheaper than databricks. Likely not

Re: Most companies do not need Snowflake or Databricks

#99

Post by bushbaba »

rglullis wrote: post #60
lr4444lr wrote: post #30

...a talk with your local Dev Ops Engineer/ manager to discuss how to secure your implementations This right here is exactly why Snowflake is a good fit for my org.: we could pay someone's salary to install, maintain, and upgrade some open source alternative (and the VPS to run it), or we could just pay for Snowflake and stop wasting engineering's time on the Data Science's team's stuff, which frees them up to move f…

> we could pay someone's salary to install, maintain, and upgrade some open source alternative (and the VPS to run it), or we could just pay for Snowflake and stop wasting engineering's time You are missing the "we could pay someone that provides a commodified, open-source alternative to Snowflake" option. It would be cheaper, it wouldn't require any of your engineering team resources and it would help us all get rid…

Would it be cheaper? Cloudera fits the bill and it’s for sure not cheaper

Re: Most companies do not need Snowflake or Databricks

#100

Post by bushbaba »

Aurornis wrote: post #43
mrweasel wrote: post #34

Most companies don't need a lot of things. I don't do a whole lot of data analytics anymore, but I'd say: Start with figuring out how much data you actually have. We still see companies claim to have vast amounts of data, but in reality they are talking about less than a TB of data, frequently just a few 100GB. When you operate at that scale, just chuck your data into whatever database you're comfortable with and do…

> We still see companies claim to have vast amounts of data, but in reality they are talking about less than a TB of data, frequently just a few 100GB. When you operate at that scale, just chuck your data into whatever database you're comfortable with and do SQL queries, it's fine. I did some work with a company that talked about their “big data”, bragged about their data lake, and made grandiose statements about how…

You can buy a single server with TiBs of ram. That’s why I think motherduck/duckdb really shines. It’s the best solution for 99% of users whose data can be processed relatively quickly on a single machine.
Post reply on HN