Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Big data is dead

motherduck.com

Post reply on HN
Print view View original

281–290 of 444 posts

Re: Big data is dead

#281

Post by andix »

primax wrote: post #227
andix wrote: post #35

I see it all the time: people develop applications that will never ever get a database size of over 100GB and are using big data databases or distributed cloud databases. Often queries only hit a small subset of the date (one customer, one user). So you could easily fit everything into one SQL database. Using any of the traditional SQL databases takes away a lot of complications. You can do transactions, you can quer…

I think a key driver of this is not having to use SQL. I like DynamoDB and EdgeDB because I can use a more modern and reasonable language to interact with the database.

That’s a good point, I also think that there should be some modern alternative to SQL. I really like how you can query databases with LinqPad (c#) and how it renders it into a nested table tree. All relations are clickable/expandable, so if you find something interesting in your result set, you can just expand additional rows from other tables. In the background it just creates sql via an ORM, not only once I more or less copy and pasted that generated sql into a view.

But linqpad is not useful if you don’t get the pro version, only then you get code completion. So it’s not really the answer to the problem.

Re: Big data is dead

#282

Post by fuziontech »

ryadh wrote: post #277

While I get that they're sometimes useful to trigger debate, I don't really subscribe to very bold statements. We are drowning in data, it's all around us. Information overload is real. Data enables most of our daily digital experiences, from operational data to insights in the form of user facing analytics. Data systems are the backbone of the digital life. It's is an ocean and it's all about the vessel you pick to…

100% agree. One of the biggest assets we had at was the data we collected. We built models on it that would determine how markets were run and whether drivers and passengers were safe. These were key features that enabled us to bring a quality service to customers (over ye ol' taxi). The same applied to the autonomous cars, bikes, and scooters. We used data to improve placement of vehicles to help us anticipate and meet demand. It was insane how much data used to build these models.

To say big data is dead sounds to me like someone desperate for eyeballs.

I do think there is a huge opportunity for DuckDB - running analytics on 'not quite big data' is a market that has always existed and is arguably growing. I've seen way too many people trying to use Postgres for analyzing 10 Billion row tables and people booting up an EMR cluster to hit the same 10 Billion rows. There is a huge sweet spot for DuckDB here were you can grab a slice of the data you are interested in, take it home and slice and dice it as you please on your local computer. I did this just this weekend on DuckDB _and_ ClickHouse!

Disclaimer: I work at a company that is entirely based on ClickHouse.

Re: Big data is dead

#283

Post by spopejoy »

ryadh wrote: post #277

While I get that they're sometimes useful to trigger debate, I don't really subscribe to very bold statements. We are drowning in data, it's all around us. Information overload is real. Data enables most of our daily digital experiences, from operational data to insights in the form of user facing analytics. Data systems are the backbone of the digital life. It's is an ocean and it's all about the vessel you pick to…

I guess the article title is a "bold statement" but maybe the biggest insight in there is that people don't think hard enough about throwing old data away, and it hurts them. This is a liferaft for drowning in data and is more "bold" organizationally, as it actually takes a certain kind of courage to realize you should just throw stuff away instead of succumb to the false comfort that "hey you never know when you might need it".

Weirdly there's a similar thing that can happen to codebases, specifically unit tests and test fixtures that outlive any of their original programmers, nobody understands what's actually being tested and before each release lose days/weeks hammering to "fix the test". The only solution is to throw it away, but good luck getting most teams to ever do that, because of the false comfort they get -- even though that fixture is now just testing itself and not protecting you from any actual bugs.

I mean how often does Netflix need to look a viewing habits from 2015? Summarize and throw it away.

Re: Big data is dead

#284

Post by zmmmmm »

To be honest, I slightly disagree about data size. I think the big data is there to be had, the real story is that data science itself has not panned out to provide the business value that people asserted would come from it. Data volumes haven't risen more because in the end, it turns out most of the things businesses need to know are easily ascertainable from much smaller data and their ability to action even these smaller very obvious things is already saturated.

It doesn't help that we've shifted into a climate where hoarding data comes with a huge regulatory and compliance price tag, not to mention risk. But if the value was there we would do it, so this is not the primary driver.

Re: Big data is dead

#285

Post by didip »

travisgriggs wrote: post #22

I've made anecdotal observations similiar to this over the last 10 years. I work in AgTech. A big push for a while here has been "more and more more data". Sensor-the-heck out of your farm, and We'll Tell You Things(tm). Most of what we as an industry are able to tell growers is stuff they already know or suspect. There is the occasional suprise or "Aha" moment where some correlation becomes apparent, but the thing a…

I've been telling folks, storing everything all the time is wasteful, a better alternative is:

1. Keep the raw full data for short period of time, at most 1 month.

2. Downsample what you need for longer period of time (5-10% of the full data).

3. Aggregate your metrics on a yearly basis to save money and compute costs.

Re: Big data is dead

#286

Post by zmmmmm »

guardiangod wrote: post #4

There is literally a post on front page on ChatGPT, and Microsoft and Google are preparing to duke it out starting in the _next 2 days_ over big-data generated 'chat' result. Big data was never going to be useful to even medium size enterprises, unless anyone can get public access to PBs of data, but that doesn't mean big data is dead. ChatGPT is literally changing how school will test their students, for a start. Ma…

Yes, this occurred to me as well. The counter narrative here is in fact that the story of the last 2-3 years has been break throughs in AI have come about mostly by scaling up their network sizes and training data sets 5 orders of magnitude or so.

I guess the take away however is still that regular businesses really just can't play in this game and should not be assuming they have big data until that fact asserts itself out of necessity rather than the other way around.

Re: Big data is dead

#287

Post by danuker »

lostmsu wrote: post #128
danuker wrote: post #6

I agree with many of the points here. My cheap no-name old laptop SSD writes with 170MB/s. A customer has a name, address, email and order. Let's say 200 bytes for each. That means I can write 844000 new customers per second, far outside my personal marketing reach. My disk is 240GB, which means I can store data for 1.2 billion customers. It'll take a while until I become that successful.

> 170MB/s That is not random access speed. For random access my relatively high-performance SSD only does 42MB/s reading and 80MB/s writing.

Indeed there's probably some caching going on.

Re: Big data is dead

#289

Post by suction »

didgetmaster wrote: post #220
pjmorris wrote: post #90

Earlier quoted context omitted.

> I used to joke that Data Scientists exist not to uncover insights or provide analysis, but merely to provide factoids that confirm senior management's prior beliefs. So, a synonym for 'consultant?' :)

In the news business, if your story or opinion backs up the preconceived notions of the investigative reporter then you are a 'source' otherwise you are a 'conspiracy theorist'.

[dead]

Re: Big data is dead

#290

Post by papito »

First they came for the sacred microservices, now they are after Bid Data. What. Is. Happening.

Don't get me wrong, I love it. It's about time people got off these stupid and shockingly expensive bandwagons.

Post reply on HN