Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Be good-argument-driven, not data-driven

twitchard.github.io

Post reply on HN
Print view View original

141–150 of 168 posts

Re: Be good-argument-driven, not data-driven

#141

Post by bee_rider »

Shacklz wrote: post #29

> Are you prepared to do some very very fancy statistics? I'd extend this with "... while understanding what you're doing?" I've seen it so many times already, someone does some A/B-test and then presents a very fancy looking slide-deck with all kinds of crazy-looking math. But if you start to ask questions, it's all very obvious that they didn't really understood what they were doing and that very often it doesn't r…

I think "Be brutally honest about you many assumptions and caveats" at least implies that.

I mean, in an informal setting there's room for an honest person to say "well I did some math and I don't really get it but I think it says...," but I think this article is addressed to software engineers and scientists. Someone representing themself as an engineer or scientists has a professional ethical responsibility to some sort of... I dunno, epistemic honesty, the knowledge of what their expertise covers, and communicating their limitations to laymen.

The person with the A/B test in your example is either a liar because they are misrepresenting what their tool says, or they are a liar because they are misrepresenting their ability to tell you what it says, but either way they are a liar.

Re: Be good-argument-driven, not data-driven

#143

Post by bumby »

kqr wrote: post #4

While I agree completely with the premise of this article, on the other hand I'm weighing the relatively robust findings by Meehl et al. They find, time and time again, in all sorts of fields, that extremely parsimonious models like equal-weighted linear regression of one or two predictors outperform expert judgment[1]. One would think this is cognitively dissonant enough, but it gets worse: This article, with the th…

>Some things are reliably measurable; for those things be data-driven. Other things not so much, then use your expertise.

Maybe too much of a nit-pick, but how does one build expertise without data? I'll grant that it may be informally or subconsciously collected but it's still data.

It makes me think of Malcolm Gladwell's book Blink. There are lots of experts who can subconsciously chunk data to make intuitive and reliable decisions. But they got to that point often gathering lots of data in the form of experience.

Re: Be good-argument-driven, not data-driven

#144

Post by TimTheTinker »

wtetzner wrote: post #129
TimTheTinker wrote: post #120

Earlier quoted context omitted.

> People tend to follow incentives, rather than focusing on making wise decisions. This is the key issue. Solving it isn't easy -- it requires people who are wise, and wisdom is a scarce commodity.

Even wise people likely follow the incentives. What is wise about doing something that your employer doesn’t reward in exchange for doing something that they will reward?

It's wise to do what's morally right, regardless of the consequences.

Re: Be good-argument-driven, not data-driven

#145

Post by cbtacy »

ekianjo wrote: post #42
shubb wrote: post #35

The related problem that I see actually more often is the "you don't have big data" problem. You know, in data science, you see people spending hours writing pandas scripts that replicate a few clicks in excel for a one of analysis. You see datasets of a few gigabytes being processed with spark when SQL would be fine. You see ML techniques being thrown at questions that could be answered simply and reliably with basi…

Talking to people is not going to help you either. You end up getting a lot of noise and making sense of what you hear is difficult. When you keep probing you will get to hear stuff thats not really critical and just often made up because you ask too many questions. Classical trap of market research.

Very dated thinking. Suggest you read up on Lean Customer Development (for example).

Re: Be good-argument-driven, not data-driven

#147

Post by throw_nbvc1234 »

int_19h wrote: post #139
klenwell wrote: post #117

Earlier quoted context omitted.

> They find, time and time again, in all sorts of fields, that extremely parsimonious models like equal-weighted linear regression of one or two predictors outperform expert judgment. I came across this in Thinking Fast and Slow. Kahneman was a big fan of Meehl and restates the point: The important conclusion from this research is that an algorithm that is constructed on the back of an envelope is often good enough t…

It seems to imply that expertise-driven design gave us Vista and Win7 while the data-driven one gave us Win8, Win10, and Win11. It's notable that, from this list, Win7 seems to be the only one that people genuinely liked.

Define "people". Tech people, people/customers in general, some other group such as shareholders? Both your point and the point your responding to could be true at the same time both anecdotally and/or in the data. Anecdotes are probably just another form of "expert opinions"

Re: Be good-argument-driven, not data-driven

#148

Post by iamcurious »

int_19h wrote: post #139
klenwell wrote: post #117

Earlier quoted context omitted.

> They find, time and time again, in all sorts of fields, that extremely parsimonious models like equal-weighted linear regression of one or two predictors outperform expert judgment. I came across this in Thinking Fast and Slow. Kahneman was a big fan of Meehl and restates the point: The important conclusion from this research is that an algorithm that is constructed on the back of an envelope is often good enough t…

It seems to imply that expertise-driven design gave us Vista and Win7 while the data-driven one gave us Win8, Win10, and Win11. It's notable that, from this list, Win7 seems to be the only one that people genuinely liked.

Yup, it seems a side effect of data driven approach is that Windows no longer cares about its own reputation.

Re: Be good-argument-driven, not data-driven

#149

Post by crabmusket »

guerrilla wrote: post #24
crabmusket wrote: post #20

Earlier quoted context omitted.

The pragmatists went a little bit too far in my opinion, though it has been a long time since I read any of them. Popper is describing observations, not reality. I highly recommend Conjectures if you can find a copy. It's a short read and interesting.

What do you mean that they went too far? James and Peirce were not describing "reality" (in this discussion anyway. [1][2]) but rather were instrumentalists and thus saw every theory as having a purpose. That's the whole point of the squirrel argument. It not just "depends on what you mean" (as per analytic and some medieval philosophy) but also depends on what you're trying to do (which in turn depends on what you w…

I think I'm out of my depth at this point actually, and maybe shouldn't have opined as readily as I did on the pragmatists. I did a little recap of where I'd encountered the pragmatists before and realised I only read Royce, who was a friend and interlocutor of James. But I don't think he could be called a member of the pragmatist school, so I shouldn't take the impressions I got from him to be representative!

The impression I had of pragmatism was that it made claims about absolute truth or reality. That's where I felt things were taken a little too far. But the impression I have may be a caricature or misunderstanding on my part.

Deutsch has a fair bit to say about instrumentalism in Beginning of Infinity which I will leave to the interested reader to discover.

Re: Be good-argument-driven, not data-driven

#150

Post by borski »

In Range, David Epstein talks about about NASA and some of their disasters, like the explosion of Challenger. NASA is the entirely encased in specialized knowledge, and has a completely data-driven mindset, with no room for logic. If you can't prove it with data, they wouldn't even consider it. He explains that, “Reason without numbers was not accepted. In the face of an unfamiliar challenge, NASA managers failed to drop their familiar tools... The Challenger managers made mistakes of conformity. They stuck to the usual tools in the face of an unusual challenge.” Even though the mistake that led to the Challenger disaster could have been caught, it was the uniformity of thinking that lead to an organizational blind spot, and that uniformity was to be too focused on data-driven arguments.

There is a famous call prior to the disaster on which engineers had raised the concerns but it was based on intuition and a few cherry picked samples, not a full set of data, and this was the night before the launch. Because of the lack of data, they went ahead with it and we all know the tragedy that ensued. Moreover, other engineers who agreed that there was an issue didn't speak up, because they too lacked the data, and knew that management wouldn't care.

Post reply on HN