Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Big data is dead

motherduck.com

Post reply on HN
Print view View original

261–270 of 444 posts

Re: Big data is dead

#261

Post by blakeburch »

alexpetralia wrote: post #5

I am writing an essay series on this topic: last-mile analytics and how an abundance of data must be ultimately converted into (measurably correct) action. If anyone wants to follow along, the series is here! https://alexpetralia.com/2023/01/19/working-with-data-from-s...

That looks like a huge undertaking, but kudos for taking the time. I'll be following along. Totally agree that all data should be tied to the business value that it's driving.

Unfortunately, I've found that many data teams focus more on making the data clean and available. They never drive the conversation about what actions are being taken with the data. That leads to them being treated as cost centers. Wrote a similar post about my perspective on it - https://bytesdataaction.substack.com/p/transform-your-data-t...

I'd love to chat about the space more with you if you're interested! Email in bio.

Re: Big data is dead

#262

Post by gesman »

Customer pays data analytics vendor to tackle bunch of their [low quality, big size] data.

If you have no tangible capabilities to do above, asking customer "ARE YOU IN THE BIG DATA ONE PERCENT?" will be the quickest way out of the door.

Re: Big data is dead

#263

Post by SkyBelow »

slt2021 wrote: post #36
fdgsdfogijq wrote: post #14

"For more than a decade now, the fact that people have a hard time gaining actionable insights from their data has been blamed on its size." The real issue is that business people usually ignore what the data says. Wading through data takes a huge amount of thought, which is in short supply. Data Scientists are commonly disregarded by VPs in large corporations, despite the claims about being "data driven". Most corpo…

I used to joke that Data Scientists exist not to uncover insights or provide analysis, but merely to provide factoids that confirm senior management's prior beliefs. I did several experiments, and noticed that whenever I produced analysis that was in line with what management expected - my analysis was praised and widely disseminated. Nobody would even question data completeness, quality, whatever. They would pick so…

This isn't just "Data Scientist" but scientist as well. The more a finding is in contradiction, either with existing scientific consensus or even with just popular culture, the more the science is criticized. I've seen unequal criticism based on how much people wanted the results to be true/false and even after responding to the criticism I've seen people just ignore science they don't like.

The skepticism isn't a problem, the unequal application of it, the potential to harm careers, and the chilling effect as people wisen to how best meet their own personal goals is.

Re: Big data is dead

#264

Post by ryguyrg »

PeterisP wrote: post #188
travisgriggs wrote: post #22

I've made anecdotal observations similiar to this over the last 10 years. I work in AgTech. A big push for a while here has been "more and more more data". Sensor-the-heck out of your farm, and We'll Tell You Things(tm). Most of what we as an industry are able to tell growers is stuff they already know or suspect. There is the occasional suprise or "Aha" moment where some correlation becomes apparent, but the thing a…

Fine-grained measurement is useful when you have options for fine-grained action. You don't need a chip to tell you that the soil is dry, but if you can use that chip to regulate drip irrigation that can apply substantially different flow to different plants, then you can get a not-too-much, not-too-little watering even if you have a big variation in conditions. You don't need a big analysis to acknowledge that every…

Agreed. But how many executives will agree to take these fine-grained actions to achieve value from the data? How many data teams are able to build up a strong-enough argument to convince them?

I've worked on many product-led-growth initiatives in the software industry. The software industry is probably the biggest 'believer in data' there is -- many scientific-forward minds who understand the value. However, even in the software industry, it's really hard to convince folks that if you make 5 improvements that net 1% conversion gain each, you can dramatically improve revenue.

Re: Big data is dead

#265

Post by slaymaker1907 »

0xdeadbeefbabe wrote: post #154
0cf8612b2e1e wrote: post #112

Earlier quoted context omitted.

Teachers assign better scores to papers with better penmanship. I forget how strong the effect was, but using a keyboard does help equalize some biases.

Why equalize that bias?

Because there are a plethora of disabilities which make neat penmanship difficult.

Re: Big data is dead

#266

Post by ryguyrg »

snarf21 wrote: post #104
guardiangod wrote: post #4

There is literally a post on front page on ChatGPT, and Microsoft and Google are preparing to duke it out starting in the _next 2 days_ over big-data generated 'chat' result. Big data was never going to be useful to even medium size enterprises, unless anyone can get public access to PBs of data, but that doesn't mean big data is dead. ChatGPT is literally changing how school will test their students, for a start. Ma…

Big Data drives the most profitable and society bending changes of all time, just to serve us better Ads.

Okay, Google as a company and as a product is definitely in the top 1%, or top 0.0001% where big data drives profitable and bending changes ;-)

Re: Big data is dead

#267

Post by threeseed »

spaintech wrote: post #216
threeseed wrote: post #194

Earlier quoted context omitted.

I assume you've never actually worked at a bank. They've been working to implement your ideas for decades and none of it requires LLMs or any machine learning techniques. Basic old ETL is more than sufficient. The issue is that (a) the calculations they need to perform are complex and take time to run (b) there are financial regulations that weave its way through those system and (c) there is a lot of legacy code esp…

Well, that is an understatement. I do agree with you that banks have been trying to fix decades old application. But in this process, you don’t need ETL, nor all the process and development to accomplish these ideas. Conceptually the idea builds its self (it learns) how to threat the data, quite revealing and near real time. Considering you account for security and privacy, then you basically shift your input into th…

So you're not actually talking about back-ends system but about the front-end.

In that case, chat-bots have existed for years and consumers largely don't like them.

In your scenario you can transfer money in a few clicks rather than having to write out an entire conversation.

Re: Big data is dead

#268

Post by mritchie712 »

olyjohn wrote: post #254
mritchie712 wrote: post #72

Earlier quoted context omitted.

Size isn't the real problem, it's time. Are you going to take the time / money to set up a warehouse, get all the data into with an ETL product, set up dbt or some other transformation layer, set up a BI tool and build the reports and dashboards, etc. Regardless the size of your data, you still need to get it in one place and model it in a way it's actually usable.

Can't we just give it to that one IT guy down in the basement?

Hey, I used to be that guy (and still am).

Re: Big data is dead

#270

Post by blakeburch »

mikepk wrote: post #204

We need to re-think how to make data _useful_. The fact that the value hasn't materialized after decades of attempts, billions of dollars, and lots of tools and technology points to the fact that our core assumptions and patterns are wrong. This post doesn't go far enough. It challenges the assumption that everyone's data is "big data" or that every company's data will eventually grow to be big data. I agree that "bi…

The problem is that no tool alone can make data useful. It requires human ingenuity to come up with a theory, gather the required data, then test and verify the theory.

We've gotten to a point where the first and last step get skipped. Business leaders see other companies doing interesting things with data, so the answer must be "gather all the data"! Internal teams end up focused on gathering the data without the context of how it might be used.

We need to train data teams to not focus on the data as the product. Instead, they should be responsible for driving business actions. Gathering and cleaning the data should just a byproduct of that activity.

Post reply on HN