Goodbye, data science

I think this hits the nail on the head. People are learning the meritocracy they were taught growing up isn't real so why would they work 80 hour weeks for a 25% bonus instead of a 10-15% bonus. The calculus gets even worse when the bonus people get is insignificant to that of switching careers often.

For what you specifically experienced, my opinion, the bigger the organization the more inevitable this seems to become. To make things worse the size of the organization isn't limited to just a company or non-profit but to the size of all groups involved, i.e. a small charity or non-profit that's part of a huge government program is similar to a small engineering team in a huge tech company. They could do huge things or be completely worthless and so long as they pass along positive messages up the chain and the org or company as a whole is doing well then yay no consequences.

We're (hopefully) at the beginning of a cycle where companies realize they are causing apathy amongst the majority of the employed and hopefully experiment (and succeed) in providing meaningful pay raises to the lower echelons which will come at the short term costs of profits but are justified for long term productivity. Or we'll just keep divolving into a dystopia

You should pay the price for data leakage very quickly in production.

Does management look at slides or AB test dashboards?

> One of the things I don't like about statements like this said in a Data Science context, is that they are true outside of Data Science as well. Executives make big decisions, managers make smaller decisions, nobody can evaluate how good/bad they really were for months or years. Engineers build something amazing, or build a house of cards, nobody cares as long as the money people are happy, even if the business use case turns out to be wrong in the long run.

This is purely anecdata, but I have found that this is more pronounced in a data science context. Managers and executives are (in my experience) more willing to admit they don't understand engineering work product and seek input from technical advisors, and executives and managers deal with decision making on a daily basis and understand that it can be nuanced. But since almost everyone reads financial reports or has to make a chart in Excel every now and then, they know enough to read someone else's analysis but not enough to recognize their knowledge gaps (particularly wrt advanced statistics).

The problem is that nobody actually wants data science. They want data pseudoscience.

And for the same reason that people tend to want pseudoscience instead of science in any other domain, too. Science is slow, tentative, and messy, and usually responds to questions with even more questions rather than with answers.

Pseudoscience tends to be much more concerned with exuding confidence and providing clean-cut answers. It's what happens when a desire for science meets a need for instant gratification. Along the way, things like blinding and controls and watching for bias and validating assumptions tend to get dropped when they're inconvenient or difficult to explain. And they're always inconvenient and difficult to explain.

I don't know how replicable my success is. but, depending on where you work, it may be that as a data scientist you can provide more value to your business purely using your software skills than with any kind of stats knowledge, by figuring out how to unfuck existing crufty bureaucratic workflows. this can be more directly useful than any amount of hyperparameter twiddling on some ridiculous neural network chimera. at my last job I could see so much of people's time wasted on fucking idiocy and my mind rebelled against it, I had this drive to rip it all out and Do It The Right Way(tm). and in doing that, I learned a lot about software development, tooling, version control, documentation, and so on. one thing led to another, and I had turned myself into a software engineer.

nowadays I do -- well, fudging slightly but you could describe it as "industrial automation control". writing libraries to provide convenient abstractions for controlling industrial equipment, writing robust scripts to drive that equipment, run physical tests on the $widgets we make, aggregate the experimental data, store it, etc. in the interviews they liked how (in my DS job) I had taken existing inefficient excel based workflows that had human-in-the-loop, and automated them, made unit tests, wrote docs, considered failure modes that nobody had considered before, things like that. and I just read about a fuckton of different stuff. for example in the interviews they wanted to know if I had worked with concurrency, I said I hadn't because it just didn't come up in the work I did. but I knew a little about it because I read voraciously, then I was able to answer all the theoretical questions they posed about locks and threads and async and so on. obviously that didn't mean I really knew about concurrency (that's a kind of deep metis that can only be acquired by practical experience and I'm still only scratching the surface of it), but it demonstrated that I had curiosity to learn about the field outside of the immediate things I worked on day to day.

during that job hunt I also had a strong offer from a company that wrote software for the visual effects industry and they wanted someone to improve their automated testing and continuous deployment frameworks. I didn't know much about CI but I knew about testing (pytest and hypothesis and things like that). they liked me talking about that kind of thing.

I guess the lesson is, if you are right now a data scientist and you want to be a software engineer, you can just decide to be that right now. be proactive and find a software problem to solve, and solve it. you don't have to ask permission to do this .. what are they going to do, tell you to stop being useful? note what you did, then figure out how to do the next thing better based on what you learned. your pay stub will say you're a data scientist, but you should just think of it as clandestine self-directed on-the-job training for your next job, so you can talk about it in the interviews. does that make sense?