I worked as an engineer on an academic project that used sound to detect chewing. I acted as a domain expert to help the grad student doing the project understand how to select an appropriate microphone to couple to skin, how to interface that microphone to an ADC, offered insights into useful features to explore based on human biology, found a sound processing library that gave us 100's of new features for very little work, helped design a portable version of the electronics, did purchasing, helped troubleshoot during the times we were getting poor results, monitored human safety so we weren't connecting student volunteers to earth ground or AC power, helped write up human subjects committee review application forms, tracked down sources of electrical noise in the system, helped analyze results, reviewed 3D models for a headset to hold the microphone and electronics, wrote some signal processing/feature extracting code, selected an appropriate SBC system, explored confounding factors that were different between lab tests and field tests, and more. The grad student organized tests to record chewing sounds and ground truth video (a cheap camera in the bill of a baseball hat), arranged for a Mechanical Turk-like service to mark up the sound data based on the ground truth using a video analysis app I found, tried different pre-written ML algorithms to figure out which worked the best across different types of food, and analyzed which features contributed most to accuracy so we could eliminate most of them and get the thing to run on a wearable system with reasonable battery life. Then we collaborated with some medical doctors (our target customer for the device, it was not intended for consumers) to run tests on kids in their eating labs and wrote papers about it. The project ended when the grad student graduated.
I worked on other ML projects as well. A system that analyzed the syslogs of dozens of computers to look for anomolies. I wrote the SQL (256 fields in the query! Most complex SQL I've ever written) that prefiltered the log data to present it to the ML algorithm. And built a server that sniffed encrypted log data we broadcast on the local network in order to gather the data in one place continuously. Another system used heart rate variability to infer stress. I helped design a smartwatch and implemented the drivers that took in HRV data from a Bluetooth chest strap. We tested the system on ourselves. None of our ML projects involved writing new ML algorithms, we just used already implemented ones off the shelf. The main work was getting the data, cleaning the data, and fine tuning or implementing new feature extractors. The CS people weren't familiar with the biological aspects (digging into Grey's Anatomy), sensors, wireless, or electronics, so I handled a lot of that. I could have done all the work, it's not hard to run ML algorithms (we often had to run them on computing clusters to get results in a reasonable amount of time, which we automated) or figure out which features are important, but then the students wouldn't have graduated :-) Getting labeled data for supervised learning is the most time consuming part. If you're doing unsupervised learning then you're at the mercy of what is in the data and you hope what you want is in there with the amount of detail you need. Interfacing with domain experts is important. Depending on the project a wide variety of different skills can be required, many not related to coding at all. You may not be responsible for them all, but it will help if you at least understand them.