Earlier quoted context omitted.
The problem is that nobody actually wants data science. They want data pseudoscience. And for the same reason that people tend to want pseudoscience instead of science in any other domain, too. Science is slow, tentative, and messy, and usually responds to questions with even more questions rather than with answers. Pseudoscience tends to be much more concerned with exuding confidence and providing clean-cut answers.…
> The problem is that nobody actually wants data science. They want data pseudoscience. Technically, I think investors & owners would want the company to use real data science to improve products & maximize profits. Everybody in the middle just wants to use data to lie to get promoted faster - because you don't get promoted for actually doing a good job - you get promoted for convincing people you did a good job, and…
This is based on the assumption that companies are focused on long term profits and stability, and I’m not sure why anyone believes that to be the case anymore. The vast majority of companies are run based on next quarter’s stock price or growth metrics.
I worked on a newly formed data science team coming out of grad school that was tasked with taking some predictive initiatives that the company had relied on external consultants to produce, and implementing them in-house. The external team’s results always looked exactly like what the business wanted to hear, but they rarely played out in practice. This was in part because the underlying data quality was terrible, and the company wasn’t executing in a way that allowed anyone to actually answer the questions being asked. The consultants would just torture the data until they could come up with a report that would ensure the company would come back the following year. So we spent a lot of time trying pouring cold water into the business groups who saw data science as a magic wand that would conjure up more money at no cost. But we never were able to convince them to invest in anything that would take longer than a year. Anything that would require a change in their marketing or strategy executions that wouldn’t immediately deliver increased results was just a non-starter. But actual data science requires that kind of investment for long-term layoffs. So the data science team became figure-heads, never given the buy-in to actually make impact on business, but kept around so teams and leaders could tout being “data-driven” and throw “AI” and “machine-learning” into PR and marketing materials.
You aren’t wrong about middle management is looking to get promoted faster. But every single individual from the employee looking for a promotion to the executive suite to the investors are addicted to incentive windows no longer than 6-12 months.