Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Big data is dead

motherduck.com

Post reply on HN
Print view View original

341–350 of 444 posts

Re: Big data is dead

#341

Post by wowJustwow »

Interesting take from a Googler.

Big data hype never felt to me like anything more than a hype campaign to help big tech research ML/AI.

Larry even rambled as much: https://arstechnica.com/information-technology/2013/05/larry...

It appears not all Googlers got the memo.

Everyone else is in the way of him solving big problems! Not like such work could not be distributed among technologists and researchers around the globe via the internet. Help Google do it!

I am leaning into “Deep Work” going forward; will slowly iterate on my own model creation and collaborate with like minded folks. I’m fucking done with intentionally empowering billionaire minority who convinced an ignorant political gerontocracy that minority is capable of magic.

Anyone prattling on with common tropes of “longtermism”; nation state nutters, religious, technocrats; are appealing to non-existent authority they see a a magical future for us! Give them your money to insure it arrives! They have zero ability to insure such outcomes and a lot of upside to making people believe such today.

Re: Big data is dead

#342

Post by fatneckbeard »

HybridCurve wrote: post #332
slt2021 wrote: post #36

Earlier quoted context omitted.

I used to joke that Data Scientists exist not to uncover insights or provide analysis, but merely to provide factoids that confirm senior management's prior beliefs. I did several experiments, and noticed that whenever I produced analysis that was in line with what management expected - my analysis was praised and widely disseminated. Nobody would even question data completeness, quality, whatever. They would pick so…

Authoritarian types consider any information derived by science which is contrary to their position as invalid or irrelevant because facts challenge their authority and ability to exercise control.

yes. I used to think the Church had a honest disagreement with Galileo about heliocentricity. When I grew up I realized the Church never cared about orbits at all, what they care about is maintenance of status quo.

And then when I got old, I realized, there is even a reason that some people want status quo... because they have usually been around long enough to see society fall apart into anarchy and mass murder, so in their mind, they are doing the right thing.

Re: Big data is dead

#343

Post by zX41ZdbW »

vgt wrote: post #295
fuziontech wrote: post #282

Earlier quoted context omitted.

100% agree. One of the biggest assets we had at was the data we collected. We built models on it that would determine how markets were run and whether drivers and passengers were safe. These were key features that enabled us to bring a quality service to customers (over ye ol' taxi). The same applied to the autonomous cars, bikes, and scooters. We used data to improve placement of vehicles to help us anticipate and m…

Didn't know that Posthog is based on CH these days. Interesting!

Check the list of companies using ClickHouse: https://clickhouse.com/docs/en/introduction/adopters/

Re: Big data is dead

#344

Post by selestify »

peatmoss wrote: post #258
slt2021 wrote: post #36

Earlier quoted context omitted.

I used to joke that Data Scientists exist not to uncover insights or provide analysis, but merely to provide factoids that confirm senior management's prior beliefs. I did several experiments, and noticed that whenever I produced analysis that was in line with what management expected - my analysis was praised and widely disseminated. Nobody would even question data completeness, quality, whatever. They would pick so…

> Data Scientist's job is to launder management's intuition using quantitative methods Ouch. This is savage, but sadly correct in many cases. HOWEVER, to play devil's advocate here, I've also seen corporate data scientists overstate the conclusions / generalizability of their analysis. I've also seen data scientists fall prey to believing that their analysis proves would should be done, rather than what is likely to…

What does a “normative” lens mean?

Re: Big data is dead

#345

Post by anthonygd »

ProjectArcturis wrote: post #245
m463 wrote: post #99

Earlier quoted context omitted.

I wonder what a data scientist could really find out about executive (over?) compensation. employee compensation. working from home. office cubicle size and layout. tool expenditure for employees vs productivity.

How would you measure productivity at scale?

Well, obviously you ask the CEO.

Re: Big data is dead

#346

Post by selestify »

CliffStoll wrote: post #317
taftster wrote: post #237

This posting was great. Highly recommended reading through. It gets really good when the author hits "Data is a Liability". > An alternate definition of Big Data is “when the cost of keeping data around is less than the cost of figuring out what to throw away.” This is exactly it. It's way too hard to go through and make decisions about what to throw away. In many respects, companies are the ultimate hoarders and can…

In a larger sense, it's a challenge to throw away stuff, just as it's difficult to trim big data. As I reach retirement, our attic, bookshelves, and cabinets must be trimmed -- and each item requires attention and a decision. Some things in the attic are obvious liabilities (what to do with a mercury barometer? A radium dial pocket watch? Old electronics?) Disposing of other stuff requires time, insight, and a sense…

Where is this excerpt from? Did you write it yourself?

Re: Big data is dead

#347

Post by RA_Fisher »

bigbluedots wrote: post #336
RA_Fisher wrote: post #334

Earlier quoted context omitted.

I think the recent industry layoffs reflect that in part. Though I think there’s a better way— that is executive data science, what I do at Zapier. The key is that I’ve built up a huge amount of econometric and economic / business skills that I apply to affect good change in collaboration with company leaders. It allows me to work with Execs using sensible analysis. I improve growth and output by helping us catch err…

That sounds amazing, what does your day to day consist of?

I get to apply my causal inference and measurement skills to a lot of compelling situations in collaboration with some really intelligent and skilled people. I truly feel like I get to perform technically excellent work, and I feel very privileged in that way.

My day to day is mostly economic / econometric analysis, conceptualization and formulation of technology, and writing, but also occasional meetings to gather information, feedback, answer questions, teach and collaborate with others.

I try to effectively “sample” the org by interacting with people up and down the org, so I can assist Execs in incorporating directors, managers and ICs knowledge / pool good information / systematize planning, etc.

Re: Big data is dead

#348

Post by DriverDaily »

disqard wrote: post #316
taftster wrote: post #237

This posting was great. Highly recommended reading through. It gets really good when the author hits "Data is a Liability". > An alternate definition of Big Data is “when the cost of keeping data around is less than the cost of figuring out what to throw away.” This is exactly it. It's way too hard to go through and make decisions about what to throw away. In many respects, companies are the ultimate hoarders and can…

> if your data can fit into RAM, what exactly are we talking about Big Data for? That's a fantastic point, and I keep mentioning the COST paper to anyone who cares: https://www.usenix.org/system/files/conference/hotos15/hotos...

COVID was proof The Right Data is better than Big Data. All those data sources to measure how many sick people we have and it turns out we just need one: Wastewater.

Re: Big data is dead

#349

Post by deterministic »

slt2021 wrote: post #36
fdgsdfogijq wrote: post #14

"For more than a decade now, the fact that people have a hard time gaining actionable insights from their data has been blamed on its size." The real issue is that business people usually ignore what the data says. Wading through data takes a huge amount of thought, which is in short supply. Data Scientists are commonly disregarded by VPs in large corporations, despite the claims about being "data driven". Most corpo…

I used to joke that Data Scientists exist not to uncover insights or provide analysis, but merely to provide factoids that confirm senior management's prior beliefs. I did several experiments, and noticed that whenever I produced analysis that was in line with what management expected - my analysis was praised and widely disseminated. Nobody would even question data completeness, quality, whatever. They would pick so…

Bingo. Welcome to the world of office politics!

Re: Big data is dead

#350

Post by makeitdouble »

rxhernandez wrote: post #238
jacobn wrote: post #117

Earlier quoted context omitted.

This sounds a lot like how my kids will listen to a teacher/coach, but not their parents...

Which is similar to how a lot of parents won't listen to their kids but will listen to the coach, teacher, or priest.

Parents can be listening to their kids 99% of the time it will be transparent and uneventful. When there’s confrontation/divergence in opinion, by definition it didn’t work out through the usual channel, and of course a third party weighting in the balance will have visible effects.
Post reply on HN