Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Big data is dead

motherduck.com

Post reply on HN
Print view View original

231–240 of 444 posts

Re: Big data is dead

#231

Post by LeanderK »

aicharades wrote: post #222
fdgsdfogijq wrote: post #14

"For more than a decade now, the fact that people have a hard time gaining actionable insights from their data has been blamed on its size." The real issue is that business people usually ignore what the data says. Wading through data takes a huge amount of thought, which is in short supply. Data Scientists are commonly disregarded by VPs in large corporations, despite the claims about being "data driven". Most corpo…

deleted- length

of course it won't and it's really ironic that you reply with the next hype

Re: Big data is dead

#232

Post by angry_moose »

nerpderp82 wrote: post #219
angry_moose wrote: post #217

Earlier quoted context omitted.

My experience with "Big Data" is it was something that couldn't be handled in a spreadsheet or on their laptop using R because it was so inefficiently coded. I got sucked into "weekly key metric takes over 14 hours to run on our multi-node kubernetes cluster" a while back. I'm not sure how many nodes it actually used, nor did I really care. Digging into it, the python code ingested about ~50GB of various files, made…

That is one of the selling points of Hadoop, you can write garbage code and scale your way out of any problem, turning the $$$ knob up to more nodes.

Yeah, that's why I got involved (I was infrastructure at the time) - how can we throw more hardware at it as the kubernetes setup they had wasn't cutting it.

One of the "data scientists" point blank said in a meeting "My time is too valuable to be spent optimizing the code, I should be solving problems. We can always just buy more hardware".

Admittedly the last little bit of analysis was pretty cool, but >>99% of that runtime was massaging all of the data into a format that allowed the last step to happen.

Re: Big data is dead

#233

Post by snapetom »

fdgsdfogijq wrote: post #14

"For more than a decade now, the fact that people have a hard time gaining actionable insights from their data has been blamed on its size." The real issue is that business people usually ignore what the data says. Wading through data takes a huge amount of thought, which is in short supply. Data Scientists are commonly disregarded by VPs in large corporations, despite the claims about being "data driven". Most corpo…

Agree on the size not being the issue. I transitioned from a data engineering manager to a data product manager at a new company. You know how much data a typical customer generates? Less than 1TB a year.

I told my VP that the engineering foul ups in the current product are easily fixable. Standard tooling and patterns exist to re-architect and solve the bottlenecks. What is much harder is a data architect to make sense of the complex data and make sure there is good value for our customers.

Guess what position I don't have on the team, and won't have due to budget issues.

Re: Big data is dead

#234

Post by alluro2 »

I'm quite surprised with data sizes mentioned in the article, and wondering if I'm missing something...We are a very small 2yo company, handling route optimization and delivery management / field service. Even with our very small number of customers, their relatively small sizes (e.g. number of "tasks" per day), being very early in development in terms of data that we collect - our database containing just customer data for 2 years is ~100GB. Which I previously considered small, and if we collected useful user metrics, had more elaborate analytics, location tracking history etc, I would expect it to be at least 3x.

We don't use any "BigData" products yet, as there wasn't any need for them, even when we provide full search and relatively nice and rich set of analytics over all the data. Yet, based on the article, we're way above most of the companies relying heavily on such tools. Confusing.

Re: Big data is dead

#235

Post by diognesofsinope »

eska wrote: post #66
slt2021 wrote: post #36

Earlier quoted context omitted.

I used to joke that Data Scientists exist not to uncover insights or provide analysis, but merely to provide factoids that confirm senior management's prior beliefs. I did several experiments, and noticed that whenever I produced analysis that was in line with what management expected - my analysis was praised and widely disseminated. Nobody would even question data completeness, quality, whatever. They would pick so…

This would match what psychologists say about humans in general: we feel first, then we use our brain to justify that feeling. We’re not rational beings.

I think the answer is simpler: people care about their careers and their family first. Think, "If the data says something that gets in the way of my career well I don't care about the data."

Had the same problem when I was an economics researcher -- publication bias for what stakeholders want to hear (often the government) is rampant because that's where funding for the economics department mostly comes from.

Re: Big data is dead

#236

Post by trieste92 »

eska wrote: post #66
slt2021 wrote: post #36

Earlier quoted context omitted.

I used to joke that Data Scientists exist not to uncover insights or provide analysis, but merely to provide factoids that confirm senior management's prior beliefs. I did several experiments, and noticed that whenever I produced analysis that was in line with what management expected - my analysis was praised and widely disseminated. Nobody would even question data completeness, quality, whatever. They would pick so…

This would match what psychologists say about humans in general: we feel first, then we use our brain to justify that feeling. We’re not rational beings.

or is it that psychologists feel that we aren't rational and use reason to justify this?

it isn't clear to me how the grounds for realizaing the theory are reconciliable with its conclusion

Re: Big data is dead

#237

Post by taftster »

This posting was great. Highly recommended reading through. It gets really good when the author hits "Data is a Liability".

> An alternate definition of Big Data is “when the cost of keeping data around is less than the cost of figuring out what to throw away.”

This is exactly it. It's way too hard to go through and make decisions about what to throw away. In many respects, companies are the ultimate hoarders and can't fathom throwing any data way, Just In Case.

Really appreciated the post overall. Very insightful.

As an anecdote to this article, when business folks have come up to me and asked about storing their data in a Big Data facility, I have never found the justification to recommend it. Like, if your data can fit into RAM, what exactly are we talking about Big Data for?

Re: Big data is dead

#238

Post by rxhernandez »

jacobn wrote: post #117
happymellon wrote: post #96

Earlier quoted context omitted.

My experience with consultants normally ended up with them asking why they are there and what report should they present to upper management. I've always used them as "independent 3rd parties" who were listened to.

This sounds a lot like how my kids will listen to a teacher/coach, but not their parents...

Which is similar to how a lot of parents won't listen to their kids but will listen to the coach, teacher, or priest.

Re: Big data is dead

#239

Post by bronson »

Guthur wrote: post #210
koolba wrote: post #143

Earlier quoted context omitted.

> if you want to succeed as a Data Scientist and be praised by management - you got to provide data analysis that supports managements ideas (however wrong or ineffective they might be). > Data Scientist's job is to launder management's intuition using quantitative methods :) It’s no different than the days when grey bearded wisemen would read the stars and weave a tale about the great glory that awaits the king if h…

And the alternative is to use the data as bones, throw it up in the air and let it tell you what to do?

Absolutely. If you don't like what K-Means is telling you, change a variable and re-run. (that's one great thing about business data: there's no shortage of variables! True, there's usually a shortage of independent variables, but fixing that is difficult and underfunded).

Re: Big data is dead

#240

Post by fijiaarone »

Somewhere along the line people were tricked into thinking that logging was data, and that to we needed to turn up every trace log to 11 on every production system.

Logs are where data goes to die.

Post reply on HN