Live data from Hacker News

Cargo cult data science

blog.richardweiss.org

31–40 of 42 posts

Re: Cargo cult data science

#31
post #3

However, that assumes that someone presenting an analytical presentation will be viewed more favourably Well, it certainly isn't helped by data scientists claiming to be better than ANY programmer and ANY statistician. Who could possibly live up to their own hype? A DS and ML winter will follow just as it did for AI.

Wat? I don't think any DS is claiming to be better than any programmer and statistician. I think the anecdote you refer to is, a DS is better at programming than a statistician and is better at statistics than a programmer. This viewpoint holds up in my experience.

I thought my ire at the term "data science" would have worn out by now, but it hasn't. To me it is a utterly meaningless term whose adoption in itself speaks volumes about the dynamics behind it.

As someone who has been doing "data science," including the programming, to me watching this trend has seemed mostly to be about hype and non-STEM-types, especially in business management and other similar areas, picking up on the importance of quantification.

I can think of two things that seem like legitimately very novel trends in my career in this area: deep learning, whose frameworks were largely abandoned in the preceding decades, and management of very large datasets. The first surprised me, the second I was talking about for years before it happened. The first seems so specialized to me, and to come after the "data science" trend, that the "data science" label seems unnecessary; the second is now usually discussed in terms of "data engineering" which I'm totally cool with.

There's a tendency to somehow suggest that the data science label is justified because statistics is all theoretical and not enough about real-world data, but that's always seemed to me to be a strawman that people erected to justify business hype labels to further their career. What it boils down to is playing off of business management's confusion that "statistics"=census numbers, counts, etc. It ignores the decades of computational statistics that was developing, and the fact that statistians are forced to deal with data as part of the field.

I wish I could find more of the papers I've read that illustrate the frustrations of statisticians and other scientists with data science. This will probably suffice, although there's more cogent, heartfelt examples: http://magazine.amstat.org/blog/2015/11/01/statnews2015/

It's difficult to describe, but for me personally it goes something like this: for years, you use R, C/C++/Python, Lisp, etc. to solve really difficult stats problems, are trying to be careful so as to not do something irresponsible. You've done work on supercomputers, laptops, you name it. Then, all of a sudden, there's an explosion of blogs, etc. talking about R, mahalanobis distances, and optimization routines as if they were discovered yesterday, by this brand new field of "data science" that's revolutionizing the world. All of a sudden because you don't know Cassandra or Spark, even though you're familiar with a lot of the underlying concepts because you've had to manage large datasets, and don't have a comp sci degree.

I don't mean ill will toward the practitioners, but it's difficult to convey what it's like to watch your field get repackaged and resold because of other peoples' misunderstandings about what it's about.

Re: Cargo cult data science

#32
post #31

Earlier quoted context omitted.

Wat? I don't think any DS is claiming to be better than any programmer and statistician. I think the anecdote you refer to is, a DS is better at programming than a statistician and is better at statistics than a programmer. This viewpoint holds up in my experience.

I thought my ire at the term "data science" would have worn out by now, but it hasn't. To me it is a utterly meaningless term whose adoption in itself speaks volumes about the dynamics behind it. As someone who has been doing "data science," including the programming, to me watching this trend has seemed mostly to be about hype and non-STEM-types, especially in business management and other similar areas, picking up…

That's fine. How do you describe a software engineer? Someone who codes? Makes APIs and tools? Handles security? Handles servers? Implements UI/UX?

So do you equally think labelling of software engineer is meaningless because it's broad?

Data science envelops many many many different sub-fields and specializations, many of them not involving any science at all, but some of them do involve science (understanding structure through observation and experimentation).

Maybe you don't like us being called "Scientists"? I can go to a journal, read research articles, and point out ones with horrible statistical analysis. Are those authors more of a scientist than I am, because they are arbitrarily in "academia"?

Finally, a dirty little secret is that the more data you have the less statistics you need. I bet even Google knows this, and their data dept. is probably the best academic statisics dept. I've ever met.

Re: Cargo cult data science

#33
This article seems to speak unfavourably towards building a basic BI infrastructure. I don't know why. There is immense value for having a single trusted source of truth where the most basic business questions can be answered ad hoc, with a suite of simple visualizations and KPIs that cover the most important facts of how the business is doing. Data like this serves as a crucial element of communication across different departments in the company.

One use of a strong BI infrastructure, that is under-appreciated, is as a sort of test suite for the business. If an important metric changes, it's extremely costly if it is not discovered very quickly. This also can lower the cost of business risks. In other words there can be much value in visible data that points to nothing new. BI isn't just a method to look for business improvements, though it certainly can be that, as well.

BI and data warehousing infrastructure doesn't replace more targeted and specific data science projects, it complements them.

Re: Cargo cult data science

#34
This is all very old stuff.

One earlier version was for AI expert systems.

Then there was object request broker architecture.

Such considerations were ubiquitous for the biggie operations research (OR) with optimization, simulation, etc. OR was so big that it was required in B-school programs.

Similarly for management science.

The lessons for how to make applications, as in this OP, were all there in the past. Indeed, operations research (OR) and management science (MS) merged to become OR/MS with a journal Interfaces that talked a lot about the points in the OP.

I went through a lot of that history and discovered lessons much like those in the OP.

> Fundamentally, to be a data driven company, data needs to be part of the internal dialogue spoken by all members.

Okay, let's stop right there! Who the heck, why, where, when did anyone ever say, argue, justify that any company should be "a data driven company"? Maybe a "market driven company", but data driven?

Really, for what kind of company should have, there is very wide agreement, from a home based business to Wall Street, and that is a money making company!

What turns on the CEO and the BoD is making money!

But not nearly all projects, data science, ..., Taylor's time and motion studies, are directly connected with making money. E.g., when I wrote software to schedule the fleet at FedEx, the main goal was just a schedule, printed out, on paper, with departure times, flight times, arrival times, etc., that would pass expert review as "flyable". Actually, saving money, i.e., optimization, was of much less interest.

> So, to avoid a cargo cult of data, organizations should stop chasing technology and start working with experienced technologists who can apply technology to solve organizational problems.

Yup.

> Executives, to understand how their project relates to company goals, and how success would be reported.

Really, reasonably well experienced problem sponsor executives will ask "Why should I do that?" and need a good answer or won't do it. Sure, one reason to do the project may be just to be playing with the latest buzz words, but most organizations have highly sensitive BS detectors that will be triggered by buzz words.

> With their bosses demanding analytical results, managers will demand analytical results from their peers, and so on, down throughout the subgroup.

Why would bosses be "demanding analytical results"? How many bosses understand good analytical results versus a lot of BS, have an accurate view of the potential of analytical results, could explain why it might be good for results to be analytical, know how to do projects that yield solid analytical results, or see how analytical results could help their careers or the goals of the company? Answer: Only a small fraction. E.g., only recently has Wall Street taken analytical results seriously for trading instead of intuitive, judgment stock picking.

> My reasoning was simple: anyone with data science on their side would be able to prove that their efforts worked better than their peers.

Then? How about the peers feel threatened and mount a gossip and sabotage campaign against the data scientist and their work? The management chain can also feel threatened.

> Basically, I had assumed a data-driven culture exists, when in reality businesses are struggling to create that culture in the first place.

They are not even "struggling to create that culture". It is a fertile, gullible imagination that believes that many organizations believe that they want "a data-driven culture".

> Data science is best viewed as a form of company culture, rather than a set of technologies.

No. Data science is best viewed as a technique, box of tools, that sometimes can, likely with work with other tools and techniques, yield some valuable results.

> I argue that it’s best to spread a data-driven culture from the top of an organization down, by requiring that reports be analytical.

Neither the spreading nor the requiring will work. Only a tiny fraction of the people in the organizations have significant ability with data science, and they will NOT make any such spreading or requiring of something they don't understand possible in the organization.

> Solutions that help measure and improve the performance of a part of the company (“we’ll help you measure marketing ROI”, or “we will introduce predictive maintenance), will spread and become enduring organizational strengths.

Not really. For "enduring organizational strengths" look to, say, high quality reasoning, writing, and presentations, powerful innovation, high determination, careful attention to the markets and the customers.

For "Solutions that help measure and improve the performance of a part of the company", that will be down somewhere near a good company Web site, good telephone courtesy, keeping lunch breaks under an hour, stopping pilfering, having good computer network management, having good computer security.

Sometimes data science, or just call it applied mathematics, and the rest of math, can mean super big bucks for a company:

Supposedly a big example is the trading software of James Simons's Renaissance Technologies.

IIRC once the CEO of American Airlines said that their subsidiary Sabre for reservations and scheduling was so important he'd sell off all the planes and just keep Sabre.

Likely the old linear programming application of the diet problem is still used effectively (i.e., save big bucks) in feed mixing for livestock, cat food, dog food, etc.

Linear and non-linear programming are likely still pillars of, worth big bucks for, operating an oil refinery.

There may be some big bucks from applying math to ad targeting on Web sites.

For large projects, the old linear programming application of "program (or project) evaluation and review technique, commonly abbreviated PERT, .... PERT was developed primarily to simplify the planning and scheduling of large and complex projects. It was developed for the U.S. Navy ..." Closely related is the "critical path method (CPM)".

https://en.wikipedia.org/wiki/Program_evaluation_and_review_...

Re: Cargo cult data science

#35
An organization executive is not a stakeholder. At best she is a leader, a formulation, an innovator. At worst she is a parasite. But not a stakeholder.

Re: Cargo cult data science

#36
post #35

An organization executive is not a stakeholder. At best she is a leader, a formulation, an innovator. At worst she is a parasite. But not a stakeholder.

In the sense that the executive has the power to enable your project for six months and risks losing $500k of stock options if she is wrong I think she is a stakeholder.

I wish I had some steak like that!

Re: Cargo cult data science

#37
post #14

When the author says: > However, that assumes that someone presenting an analytical presentation will be viewed more favourably than someone presenting something softer. Basically, I had assumed a data-driven culture exists, when in reality businesses are struggling to create that culture in the first place. I think this understanding of the situation is in itself part of the problem. It assumes that someone coming i…

> An organization which _always_ values data-driven decision making over expertise-driven decision making is always going to fall prey to this myopia. This is a huge and under-appreciated concern. It's disturbing how often success is measured by optimizing a single metric, and how resistant people can be to recognizing issues with this approach. A goal like "improve clickthrough rates" is easy to measure, but without…

>It's disturbing how often success is measured by optimizing a single metric, and how resistant people can be to recognizing issues with this approach.

this is very true. The zeal of data driven approaches sometimes reminds of "craniometry" where people tried to gauge intelligence by measuring the shape of one's skull.

The trade off of using quantitative methods is always that you might lose too much meaning. The good thing about data driven approaches is that they are transparent and enable objective decision making, but people need to pay close attention and be alert that whatever it is they are measuring still has some qualitative justification.

Re: Cargo cult data science

#38
post #4

I’d say this piece applies outside of data science, too. It’s a nice reminder that technology can lead to culture change, but cannot drive it

Technology is the defining change of our life times.

You might be surprised at how normal it is to just use technology to do the same thing faster. I'll wager most business people think of computers as glorified typewriters than can also send "memos".

Re: Cargo cult data science

#39
post #14

When the author says: > However, that assumes that someone presenting an analytical presentation will be viewed more favourably than someone presenting something softer. Basically, I had assumed a data-driven culture exists, when in reality businesses are struggling to create that culture in the first place. I think this understanding of the situation is in itself part of the problem. It assumes that someone coming i…

> An organization which _always_ values data-driven decision making over expertise-driven decision making is always going to fall prey to this myopia. This is a huge and under-appreciated concern. It's disturbing how often success is measured by optimizing a single metric, and how resistant people can be to recognizing issues with this approach. A goal like "improve clickthrough rates" is easy to measure, but without…

You preempted my followup article. I'm not sure where the balance between these two is, but I'm sure that many places get it wrong.

Re: Cargo cult data science

#40
post #14

When the author says: > However, that assumes that someone presenting an analytical presentation will be viewed more favourably than someone presenting something softer. Basically, I had assumed a data-driven culture exists, when in reality businesses are struggling to create that culture in the first place. I think this understanding of the situation is in itself part of the problem. It assumes that someone coming i…

That's a really good point, and I've seen that myopia cause problems as well. Like you say, it's about understanding the limits and applicability of data science, and how it interacts with experience and qualitative strategy.
Post reply on HN