Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cargo cult data science

blog.richardweiss.org

Post reply on HN
Print view View original

11–20 of 42 posts

Re: Cargo cult data science

#11

Post by soVeryTired »

thanatropism wrote: post #6
gaius wrote: post #3

However, that assumes that someone presenting an analytical presentation will be viewed more favourably Well, it certainly isn't helped by data scientists claiming to be better than ANY programmer and ANY statistician. Who could possibly live up to their own hype? A DS and ML winter will follow just as it did for AI.

Yes. The blogpost is about the organizational difficulties in unlocking the value of technically sound "data science" projects, but these in turn are the tip of an iceberg of "omg watson" on the executive side and "machine learning does well on $archetypal_dataset, it can do anything!" on the techie side. A while ago there was a Kaggle project to solve certain conjectures on prime number theory. Seriously?

Got a link to that Kaggle competition?

Re: Cargo cult data science

#12

Post by gipp »

gaius wrote: post #3

However, that assumes that someone presenting an analytical presentation will be viewed more favourably Well, it certainly isn't helped by data scientists claiming to be better than ANY programmer and ANY statistician. Who could possibly live up to their own hype? A DS and ML winter will follow just as it did for AI.

> Well, it certainly isn't helped by data scientists claiming to be better than ANY programmer and ANY statistician.

I don't really think that's much of a thing? I've been working in the field for several years now, and I'd say majority of my efforts when communicating with stakeholders is about _qualifying_ our capabilities and managing the expectations they're already coming in with.

Re: Cargo cult data science

#14

Post by gipp »

When the author says:

> However, that assumes that someone presenting an analytical presentation will be viewed more favourably than someone presenting something softer. Basically, I had assumed a data-driven culture exists, when in reality businesses are struggling to create that culture in the first place.

I think this understanding of the situation is in itself part of the problem. It assumes that someone coming in with an analytical presentation necessarily should be viewed more favorably than someone presenting something softer.

Coming from someone who's been working as a data scientist for several years, data-driven decision making has its limits. One very important one is a strong, strong bias towards myopic metrics (e.g. "engagement" over "lifetime value", "traffic volume" over "reputation in the market"), on the basis that they:

* Are more easily measurable

* Provide more data to work with

* Provide a stronger signal/noise ratio

* Provide much faster feedback

An organization which _always_ values data-driven decision making over expertise-driven decision making is always going to fall prey to this myopia. Fighting cargo-cult data science and building a sustainable analytical culture also means understanding the limits of data-driven decision making and that it does not replace, but supplements, "softer" expertise-driven culture.

Re: Cargo cult data science

#15

Post by Bartweiss »

thanatropism wrote: post #6
gaius wrote: post #3

However, that assumes that someone presenting an analytical presentation will be viewed more favourably Well, it certainly isn't helped by data scientists claiming to be better than ANY programmer and ANY statistician. Who could possibly live up to their own hype? A DS and ML winter will follow just as it did for AI.

Yes. The blogpost is about the organizational difficulties in unlocking the value of technically sound "data science" projects, but these in turn are the tip of an iceberg of "omg watson" on the executive side and "machine learning does well on $archetypal_dataset, it can do anything!" on the techie side. A while ago there was a Kaggle project to solve certain conjectures on prime number theory. Seriously?

> A while ago there was a Kaggle project to solve certain conjectures on prime number theory. Seriously?

I've seen a surprising number of Kaggle projects setting (or claiming to achieve) objectives that look impossible - things like extracting complex insights from such short signals that they apparently violate the pigeonhole principle.

The worst demonstration was looking at the results of a college class with "do a Kaggle project" as the final task. It was painfully obvious that all of the 'best' results were either extreme overfitting or fake data science (that is, using a strong algorithm to start and getting no gains from training).

Which means that many of the soon-to-graduate students had concluded that good data science meant getting strong results, not producing reliable and novel insights. It felt a bit like a software-centered version of what social psychology has been suffering from.

Re: Cargo cult data science

#16

Post by gaius »

gipp wrote: post #12
gaius wrote: post #3

However, that assumes that someone presenting an analytical presentation will be viewed more favourably Well, it certainly isn't helped by data scientists claiming to be better than ANY programmer and ANY statistician. Who could possibly live up to their own hype? A DS and ML winter will follow just as it did for AI.

> Well, it certainly isn't helped by data scientists claiming to be better than ANY programmer and ANY statistician. I don't really think that's much of a thing? I've been working in the field for several years now, and I'd say majority of my efforts when communicating with stakeholders is about _qualifying_ our capabilities and managing the expectations they're already coming in with.

It's quite a famous quote, I think it was the chief data scientist at LinkedIn who coined it originally.

There is real value in being a "statistical programmer" but that value can't presently be seen past the smoke and mirrors.

Re: Cargo cult data science

#17

Post by Bartweiss »

gipp wrote: post #14

When the author says: > However, that assumes that someone presenting an analytical presentation will be viewed more favourably than someone presenting something softer. Basically, I had assumed a data-driven culture exists, when in reality businesses are struggling to create that culture in the first place. I think this understanding of the situation is in itself part of the problem. It assumes that someone coming i…

> An organization which _always_ values data-driven decision making over expertise-driven decision making is always going to fall prey to this myopia.

This is a huge and under-appreciated concern. It's disturbing how often success is measured by optimizing a single metric, and how resistant people can be to recognizing issues with this approach.

A goal like "improve clickthrough rates" is easy to measure, but without some human insight it's all too easy to achieve it at the cost of overall success. Did you decrease time-on-page? Maybe your visitors feel mislead. Did you decrease conversion rate? Maybe your new visitors don't actually want your product. And so on, indefinitely, including lots of side effects you might not have convenient statistics for.

I have a depressing sensation that at least half of corporate data science consists of abusing Goodhart's Law - finding a useful metric and then naively optimizing for it until it's no longer representative of business success.

Re: Cargo cult data science

#18

Post by gipp »

gaius wrote: post #16
gipp wrote: post #12

Earlier quoted context omitted.

> Well, it certainly isn't helped by data scientists claiming to be better than ANY programmer and ANY statistician. I don't really think that's much of a thing? I've been working in the field for several years now, and I'd say majority of my efforts when communicating with stakeholders is about _qualifying_ our capabilities and managing the expectations they're already coming in with.

It's quite a famous quote, I think it was the chief data scientist at LinkedIn who coined it originally. There is real value in being a "statistical programmer" but that value can't presently be seen past the smoke and mirrors.

I think you're thinking of Josh Wills, Cloudera at the time, now Slack:

https://twitter.com/josh_wills/status/198093512149958656?lan...

> Data Scientist (n.): Person who is better at statistics than any software engineer and better at software engineering than any statistician.

A lot less braggadocious than what you're suggesting, it's just talking about it as a jack-of-all-trades type of job.

Re: Cargo cult data science

#19

Post by hkon »

soVeryTired wrote: post #13
hkon wrote: post #9

Earlier quoted context omitted.

no?

"Experiments" doesn't have four Es. Unless it's meant to be some sort of pun on "experience".

Yes, I was just having a bit of fun. People cannot even point out spelling errors without hedging their statements. (right?)

Re: Cargo cult data science

#20

Post by walterreade »

soVeryTired wrote: post #11
thanatropism wrote: post #6

Earlier quoted context omitted.

Yes. The blogpost is about the organizational difficulties in unlocking the value of technically sound "data science" projects, but these in turn are the tip of an iceberg of "omg watson" on the executive side and "machine learning does well on $archetypal_dataset, it can do anything!" on the techie side. A while ago there was a Kaggle project to solve certain conjectures on prime number theory. Seriously?

Got a link to that Kaggle competition?

Here's the link. It was a playground competition (i.e., no rewards) - "This competition challenges you create a machine learning algorithm capable of guessing the next number in an integer sequence. While this sounds like pattern recognition in its most basic form, a quick look at the data will convince you this is anything but basic!"

https://www.kaggle.com/c/integer-sequence-learning

Post reply on HN