Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Big data is dead

motherduck.com

Post reply on HN
Print view View original

271–280 of 444 posts

Re: Big data is dead

#271

Post by red-iron-pine »

Guthur wrote: post #210
koolba wrote: post #143

Earlier quoted context omitted.

> if you want to succeed as a Data Scientist and be praised by management - you got to provide data analysis that supports managements ideas (however wrong or ineffective they might be). > Data Scientist's job is to launder management's intuition using quantitative methods :) It’s no different than the days when grey bearded wisemen would read the stars and weave a tale about the great glory that awaits the king if h…

And the alternative is to use the data as bones, throw it up in the air and let it tell you what to do?

And you'd better hope the bones actually say something useful.

I was the infra lead on a data lake project and got take part all the way to breaking down the data and turning into PowerBI reports. The result was "sell more" and to clients who marketing already identified, years ago, as whales.

There were some interesting other insights, esp. w/r/t to niche products that sold around weird dates (Easter, Memorial Day, 4th July -- but not obvious gift days like Valentines or X-Mas), but it led to a lot of "you're doing it wrong!" recriminations and follow up projects.

Re: Big data is dead

#273

Post by ryguyrg »

glogla wrote: post #208
pocket_cheese wrote: post #160

Earlier quoted context omitted.

This is not true. Any column store database (bigquery, Redshift, snowflake) implements distributed compute behind the scenes. When an analyst/business intelligence people have a query return in 3 seconds instead of 15 seconds, it's actually huge. Not just in aggregate amount of time saved, but in creating a quick feedback loop in testing hypothesizes. This is especially true considering that most analyst type people…

Yup. I was just playing with some data from our manufacturing system, about 30 GB. I pulled the data to my laptop (very expensive Apple one) and while it fits on my disk just fine, it took about 15 minutes to download. I imported it to ClickHouse which took a while due to figuring out whatever compression and LowCardinality() and so on. I ran a query and it took ClickHouse about 15 seconds. DuckDB pointed to the parq…

I'd love to understand the shape of this data and some of the types of queries you're performing. It would be very helpful as we build our product here at motherduck.

I have no doubt that there are situations where the cloud will be faster, especially when provisioned for max usage [which many companies do not]. However, there are a lot of these situations even where the local machine can supplement the cloud resources [think re decisions a query planner can make].

Feel free to reach out at ryan at motherduck if you want to chat more.

Re: Big data is dead

#274

Post by blakeburch »

Great post and really resonates with my experience. Good to have some confirmation that most organizations aren't using their large swaths of data.

Although I don't think most organizations are blaming lack of actionable insights on the data size. It's the lack of prioritizing data usage over data accessibility. We need to be teaching data people business levers and teaching business people data levers.

Data should be a byproduct of an actionable idea that you want to execute. It shouldn't exist until you have that experiment in mind.

Re: Big data is dead

#275

Post by 0xB31B1B »

primax wrote: post #227
andix wrote: post #35

I see it all the time: people develop applications that will never ever get a database size of over 100GB and are using big data databases or distributed cloud databases. Often queries only hit a small subset of the date (one customer, one user). So you could easily fit everything into one SQL database. Using any of the traditional SQL databases takes away a lot of complications. You can do transactions, you can quer…

I think a key driver of this is not having to use SQL. I like DynamoDB and EdgeDB because I can use a more modern and reasonable language to interact with the database.

its really difficult to do any kind of analysis without relational queries. The standard way you do this is to have an app datastore in DDB, and an ETL job that pipes your data into some data warehouse env.

Re: Big data is dead

#276

Post by beckingz »

doug_durham wrote: post #85
0xB31B1B wrote: post #62

Earlier quoted context omitted.

There are functionally less than 1000 organizations that currently require distributed compute for data analysis. You can get off the shelf AWS units with 1000 cores, terabytes of ram and storage, etc. The cost of compute has decreased faster than the amount of data we have to store and process. What we used to do with spark jobs we can do with python on a single box.

Citations please? That's a pretty bold statement to make in the face of observed reality.

Even if this is off by two orders of magnitude and it's only 100,000 companies that need distributed compute, that means that almost all companies just need a single large computer.

Looking at the distribution of companies by employee count and assuming that data scales with employee count (dangerous assumption, but probably true enough on average), that means that companies don't need distributed compute until they get several hundred employees. [0]

[0] https://www.statista.com/statistics/487741/number-of-firms-i...

Re: Big data is dead

#277

Post by ryadh »

While I get that they're sometimes useful to trigger debate, I don't really subscribe to very bold statements.

We are drowning in data, it's all around us. Information overload is real. Data enables most of our daily digital experiences, from operational data to insights in the form of user facing analytics. Data systems are the backbone of the digital life.

It's is an ocean and it's all about the vessel you pick to navigate it. I don't believe that the vessel should dictates the size of the ocean, it's simply constrained by it's capabilities. The trick is to pick the right vessel for the job, whether you want to go fast, go far or fish for insights (ok, I need to stop pushing on this metaphor )

This visionary paper from Michael Stonebreaker (2005) predicted it quite accurately and I think is still relevant: https://cs.brown.edu/~ugur/fits_all.pdf

Databases come in various flavours and the "trends" are simply a reflection of what the current era needs

Disclaimer: I work at ClickHouse

Re: Big data is dead

#278

Post by beckingz »

threeseed wrote: post #248
fho wrote: post #139

Earlier quoted context omitted.

https://yourdatafitsinram.net/

This is such a lazy response. I/O performance is just one of many characteristics that impact performance and from experience the one you least need to worry about. RAID 0 across multiple high-end NVME drives with OS file caching is going to be more than fast enough for most use cases. The issue is running out of CPU performance and being able to seamlessly scale up/down compute with live running workloads.

A large computer is radically CPU overprovisioned for most workloads.

Re: Big data is dead

#279

Post by PeterisP »

poorman wrote: post #177

This entire post reads like "you probably don't actually have big data". What do these blockchains do that have to keep data around forever, with high throughput, and need to expose it quickly do? Are you saying they should delete parts of data in the chain? Seriously, I've spent my career working on big data systems, and while the answer is sometimes "yes you need to delete your data", I don't think that's going to…

And what about these blockchains? The full history of Bitcoin blockchain is less than 500gb, so for any analysis just getting a machine with a terabyte of RAM is both simpler and cheaper (once you include dev+ops time) than doing any horizontal scaling across multiple machines with "Big Data" approaches.

"You probably don't actually have big data" is a very valid point, not that many organizations do - most businesses haven't generated enough actionable data in their lifetime to need more than a single beefy machine without ever deleting data.

Re: Big data is dead

#280

Post by freedude »

"Among customers who were using the service heavily, the median data storage size was much less than 100 GB"

Eye-opening. Especially when combined with a recent quote from Satya Nadella, "First, as we saw customers accelerate their digital spend during the pandemic, we’re now seeing them optimize their digital spend to do more with less."

Conclusion: SaaS is easy to drop off in downturns. Just as easy as it is to buy initially.

Post reply on HN