Data Science: Reality Doesn't Meet Expectations

This article mentions that leadership often doesn't support data science, but I think it actually doesn't go far enough. Leadership doesn't just have to support the data scientists, it has to actually tell other teams to prioritize data science projects over what they are currently doing. Since these data science projects are riskier than standard projects, it makes sense that leadership doesn't often do this (and focusing on the standard projects could be the right call). However, it also means that it's very hard for data scientists to create business value.

I don't get why building a model people consider to be the "fun" part. That's mostly spitting data in, watching a loading screen, and then observing the output.

That's not fun, that's boring. The fun part is looking at the data and gleaming all these potential patterns from it, seeing what potential is there and what could be. Likewise, learning the business side and seeing what is possible no one has considered is great fun too.

My favorite part is feature engineering. Pre-processing and cleaning is fun too, but morphing the data into formats that extract a diamond from coal is a lot of fun, and what data science is all about. Clicking go on some ML algo is just icing on the cake, seeing it reveal bits maybe even I overlooked in the data.

If you like ML why not be an MLE? That's what MLEs do, and they're a more desirable job. DS is all about the research, discovering and learning new information, and making the impossible possible.

I certainly do, but I've been doing data science for the better half of a decade. It seems starting around 2015 when Data Science became a sexy title, a lot of fresh blood has been overgrown software engineers wanting the title, not knowing what they're getting into, or having faulty expectations. I don't consider this class of "data scientist" a data scientist, which is why the community has started shifting its job title away from Data Science to Research Scientist to better differentiate.

The good side of this is I'll come into a company with them expecting me to be like an over blown software engineer, and it gives me the opportunity to show off and go above and beyond what companies expect, allowing me to come off like a super hero. Though, it's definitely an uphill battle, and knowing how to work with upper management is an absolute necessity.

You can't compress information until you have it in a format that is appropriate for compression.

That is:

You can't compress (apply/create algorithms) information (data) until you have it (instrumented data collection) in a format (schema) that is appropriate for efficient compression (structured logging/cleaning).

99% of that is Data Engineering and building good engineering practices which have good data practices as a priority.

For any organization that has more than a handful of employees and more than one product, that is a non trivial task and gets more difficult the larger the organization gets.

Sometimes a young startup wants to advertise to the board, and they want you to make a presentation. I've made the mistake of showing near 100% accuracy solving a difficult problem important to the business, and expecting a strong positive result.

Instead I got a, "But, are we using deep neural networks?" type comments.

Sometimes a company just wants to market, be it to customers, or to the board. It's important to know your audience.