Big data is dead

Post by karmajunkie » Tue, Feb 07, 2023, 5:59 PM UTC

slt2021 wrote: post #36

fdgsdfogijq wrote: post #14

"For more than a decade now, the fact that people have a hard time gaining actionable insights from their data has been blamed on its size." The real issue is that business people usually ignore what the data says. Wading through data takes a huge amount of thought, which is in short supply. Data Scientists are commonly disregarded by VPs in large corporations, despite the claims about being "data driven". Most corpo…

I used to joke that Data Scientists exist not to uncover insights or provide analysis, but merely to provide factoids that confirm senior management's prior beliefs. I did several experiments, and noticed that whenever I produced analysis that was in line with what management expected - my analysis was praised and widely disseminated. Nobody would even question data completeness, quality, whatever. They would pick so…

Post by posharma » Tue, Feb 07, 2023, 6:01 PM UTC

Post by deltarholamda » Tue, Feb 07, 2023, 6:03 PM UTC

tomwheeler wrote: post #44

danuker wrote: post #6

I agree with many of the points here. My cheap no-name old laptop SSD writes with 170MB/s. A customer has a name, address, email and order. Let's say 200 bytes for each. That means I can write 844000 new customers per second, far outside my personal marketing reach. My disk is 240GB, which means I can store data for 1.2 billion customers. It'll take a while until I become that successful.

Presumably the "order" you mention is a primary key to another table, likely one that references the individual items that make up that order, so the data will be much larger than you estimate. It will grow larger still if you include web logs from your e-commerce site and event data from your mobile app so that you can correlate these orders with items that customers considered but ultimately didn't buy. How will yo…

Post by zeagle » Tue, Feb 07, 2023, 6:05 PM UTC

slt2021 wrote: post #36

fdgsdfogijq wrote: post #14

"For more than a decade now, the fact that people have a hard time gaining actionable insights from their data has been blamed on its size." The real issue is that business people usually ignore what the data says. Wading through data takes a huge amount of thought, which is in short supply. Data Scientists are commonly disregarded by VPs in large corporations, despite the claims about being "data driven". Most corpo…

I used to joke that Data Scientists exist not to uncover insights or provide analysis, but merely to provide factoids that confirm senior management's prior beliefs. I did several experiments, and noticed that whenever I produced analysis that was in line with what management expected - my analysis was praised and widely disseminated. Nobody would even question data completeness, quality, whatever. They would pick so…

Post by saltcured » Tue, Feb 07, 2023, 6:08 PM UTC

zeagle wrote: post #67

slt2021 wrote: post #36

Earlier quoted context omitted.

I used to joke that Data Scientists exist not to uncover insights or provide analysis, but merely to provide factoids that confirm senior management's prior beliefs. I did several experiments, and noticed that whenever I produced analysis that was in line with what management expected - my analysis was praised and widely disseminated. Nobody would even question data completeness, quality, whatever. They would pick so…

Huh. I've not thought of it as laundering, but I think you've basically summarised consulting in healthcare. Pay to legitimize and push through a pre-existing idea (eg let's close down a few ERs) or a delusion (e.g. lean, we don't need a waiting room) and say it was recommended by consultants to stakeholders and the public.