My entire post-college career has been successful because ML is too hard for software engineers (and scientists). Bear with me here. Long ago, during the AI winter (late 80s to 2000s) as a high school student I read about neural nets, and being interest in both biology and CS, thought that was an exciting system to learn about.
When I got to college, nobody talked about neural networks. Machine learning as a whole was considered a scurrilous science, wasting people and computer time. "There's not enough data. And the algorithms we have don't work! And even if we solved those, computers are too slow".
Fortunately, I managed to fail to get a job in the bio dept and as a consolation, was pointed at a nascent computational biology group in the CS department. I met David Haussler, then one of the few people in CS doing ML. Absolutely genius, he showed me a few papers and I tried to read them/understand them. The math was all over my head. It involved finding analytic derivatives of complicated functions. Fortunately, I was paired up with a grad student and given a reasonable project, where I downloaded all the gene sequence data for E.Coli (which wasn't even finished at the time) and managed to build a simple model of E.Coli genes and write an undergrad thesis that I only partly understood. To me, the magical part was watching gradient descent take those derivatives and update the weights.
When getting ready for my next phase of life, I was terrified that I wouldn't get a job as a programmer in Silicon Valley, because those folks all had CS, not bio degrees, and they knew how hash tables worked, and other complicated stuff that I couldn't wrap my head around. I decided, there was no chance I could afford to live in the valley in 1995 so I applied to grad school and got in; goal was to understand gradient descent.
My PhD was the most exhilirating and exhausting time of my life. Suddenly I was surrounded by people who could solve hard physics questions, understood quantum mechanics, and could come up with interesting experiments that got published in top journals. I felt like an imposter the entire time. But I fell in with a good group that encouraged me to explore things at my own pace, and I spent the next 7 years learning a ton of things, the capstone of which was understanding/converting a molecular dynamics loss function (including those painfully learned analytic derivatives) from FORTRAN to C++, and writing a gradient descent routine straight out of Numerical Recipes. I was thrilled that I could understand the magic of gradient descent but also depressed because that method doesn't solve the hard problems of biology (such as predicting protein structures de novo), and people were still saying that ML didn't work, there wasn't enough data, the algorithms sucked, and computers weren't fast enough.
That didn't sound right to me, because I knew that genetic data was exploding, and computers (especially cheap linux clusters) were changing access to computation quickly. The algorithms (circa 2001) were still mostly garbage, especially in biology. Neural networks to predict protein secondary structure had hit a wall at about 80% accuracy and nobody was doing ML to predict protein tertiary structure.
So I went back to school for a few more years because I still didn't know how to get a job in Silicon Valley. I did a 3 year postdoc with little to no machine learning, just domain-specific biology stuff that I didn't find interesting, and finally managed to get a job as a computer scientist at a national lab. It was a good pivot- I was a principle investigator, meaning I could apply for my own grants, write papers, etc, but didn't have to teach classes. I was MISERABLE! I loved the engineering, but the papers/grants/conference parts were just terrible.
But I still didn't "get" machine learning and wanted to work somewhere that did ML. I tried to get a job as a SWE at google- went through the ringer of all the hard questions, and ultimately got turned down at the last step (thanks, Larry Page) and went to work for a biotech for a year before I finally managed to get hired at Google during the "post-IPO, Google-classic" era, around 2007. My pay started rising faster than the average for the Bay Area which was a nice detail.
When I got to Google I quickly looked through all the projects doing ML and found that other than ads, there really wasn't a lot. There was rephil, and SETI, and SmartASS, none of which seemed even remotely like the ML I was interested in (deep neural networks). So I went and focused on other stuff- learning the distributed technology beneath Borg and Colossus, and mastering the google3 stack and production environments, mainly from an SRE perspective. But my job wasn't very demanding and I spent all my time writing proposals for Google to get involved in biology, because Google had distributed tech that was perfect for doing biology research.
Eventually, some senior engineer found my proposals and introduced me to the right people and I spent the next few years writing and running a large-scale distributed idle-cycle harvester that ran protein folding, protein design, drug discoveyr, and telescope design codes at large scale, while also learning large-scale data processing, because in speeding up the simulations, we were inundated with data to process. I got a few great publications out of this, and ended up being part of the cool kids club (coffee with Jeff Dean and Sanjay Ghemawat, etc) and parlayed this into a job building a new biology-specific platform vertical in Google Cloud so Google could make money off of, and improve the process of, biology research.
At some point I managed to tick off some senior person so I coudlnt' work on Research at Google, but finally- for the first time in my career- managed to land a job working full-time on machine learning- a system at google that almost nobody knows of called Sibyl. Sibyl was an innovative system that used an obscure ML concept- boosting- combining it with mapreduce- to run large-scale ML experiments that were directly part of the serving loop for Youtube, Google Play Ads, and other rapidly growing parts of the company. The profits from sibyl were enough to pay for all of google's research for several years and helped google grow tremendously. All that time I'd spent on machine learning and computer infrastructure... went to writing systems that loaded 80GB hash tables into memory just so a mapper could compute a tiny part of some gradient for some variable.
Unfortunately sibyl was actually a terrible system and I got kicked off the team for telling the leader the right way to do DL was deep neural networks on high performance computing hardware, not mapreduce on cheap linux cluster machines. I hid in a side team for years, playing around with 3d printers and other stuff, not really moving my career forward, but enjoying my job for the first time! At the same time I watched Jeff Dean finally realize that machine learning was an HPC problem (I think vincent vanhoucke managed to speed up voice recognition with 8 GPUs stuffed into a desktop) and he created TensorFlow, which still stumbled around for years before it realized it was an HPC system (see the slow transition to making more and more of the training process be parallel).
Finally, neural networks were vindicated! They solved a wide range of problems and my skills were applicable. We had the data, the algorithms, and the compute, all at once. And even better, you didn't need to be inside google to take advantage of it (except the big data, and that was changing quickly). I understand enough of the math, and the infra to finally be an ML Engineer.
But around that time I also came to a conclusion: most people working in ML are miserable. They are under intense pressure to get results a few percent better than their collaborators, and then once published, pivot to the next-next thing. Thats when I came up with one of my laws: "The very best ML models are distilled from postdoc tears". I saw a few people break down and leave the industry for good just from working on super-stressful projects where they did great work, but only reached parity with a competitor. and so I concluded: I was going to be ML-adjacent. This has been a succesful pivot for me.
What is the moral of this long story? Imposter syndrome drove me to overcome my imposter syndrome, and in doing so, along the way, I learned what I was chasing was not actually what made me happy. I'm far more satisfied puttering about using 5-year-old ML tech like object detectors to improve my microscope's ability to track tardigrades, than I am trying to become a famous researcher who unblocked the hard problems of biology. I guess that's part of the aging process and the stability that comes from having a salary so I don't have to worry if I can make rent.
ML has its own guild-like quality. There is some subgroup of ML people who will always try to move the goalposts, making the math harder, most esoteric, and less practical, while often publishing garbage until you peek under the covers and you realize they just got lucky, and scared away all the competitors with their Big Math. I wish people would stop doing this and instead focus on building relatively simple systems and not trying to chase 1% improvement in performance by making the system 3X more complicated.