Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Machine learning engineers, what do you do at work?

news.ycombinator.com

Post reply on HN
Print view View original

61–70 of 233 posts

Re: Ask HN: Machine learning engineers, what do you do at work?

#61

Post by astromaniak »

buildbot wrote: post #20
next_xibalba wrote: post #17

Earlier quoted context omitted.

Do people doing ML/DS not use conda anymore?

A lot do, personally, every single time I try to go back to conda/mamba whatever, I get some extremely weird C/C++ related linking bug - just recently, I ran into an issue where the environment was _almost_ completely isolated from the OS distro's C/C++ build infra, except for LD, which was apparently so old it was missing the vpdpbusd instruction ( https://github.com/google/XNNPACK/issues/6389 ). Except the thing wa…

It's probably easier to reinstall everything anew from time to time. Instead of fixing broken 18.04 just move to 22.04. Most tools should work, if you don't have huge codebase which requires old compiler...

Conda.. it interfere with OS setup and has not always the best utils. Like ffmpeg is compiled with limited options, probably due to licensing.

Re: Ask HN: Machine learning engineers, what do you do at work?

#62

Post by SJC_Hacker »

davedx wrote: post #5

pip install pytorch Environment broken Spend 4 hours fixing python environment pip install Pillow Something something incorrect cpu architecture for your Macbook Spend another 4 hours reinstalling everything from scratch after nuking every single mention of python pip install … oh time to go home!

Docker is your friend

or pyenv at least

Re: Ask HN: Machine learning engineers, what do you do at work?

#63

Post by naveen99 »

burnedout_dc4e3 wrote: post #52

I've been doing machine learning since the mid 2000s. About half of my time is spent keeping data pipelines running to get data into shape for training and using in models. The other half is spent doing tech support for the bunch of recently hired "AI scientists" who can barely code, and who spend their days copy/pasting stuff into various chatbot services. Stuff like telling them how to install python packages and u…

You’re living the dream. Why quit ?

Re: Ask HN: Machine learning engineers, what do you do at work?

#64

Post by copperroof »

next_xibalba wrote: post #17
davedx wrote: post #5

pip install pytorch Environment broken Spend 4 hours fixing python environment pip install Pillow Something something incorrect cpu architecture for your Macbook Spend another 4 hours reinstalling everything from scratch after nuking every single mention of python pip install … oh time to go home!

Do people doing ML/DS not use conda anymore?

If they are they should stop.

It causes so many entirely unnecessary issues. The conda developers are directly responsible for maybe a month of my wasted debugging time. At my last job one of our questions for helping debug client library issues was “are you using conda”. And if so we just would say we can’t help you. Luckily it was rare, but if conda was involved it was 100% conda fault somehow, and it was always a stupid decision they made that flew in the face of the rest of the python packaging community.

Data scientist python issues are often caused by them not taking the 1-3 days it takes to fully understand their tool chain. It’s genuinely quite difficult to fuck up if you take the time once to learn how it all works, where your putbon binaries are on your system etc. Maybe not the case 5 years ago. But today it’s pretty simple.

Re: Ask HN: Machine learning engineers, what do you do at work?

#65

Post by bentt »

burnedout_dc4e3 wrote: post #52

I've been doing machine learning since the mid 2000s. About half of my time is spent keeping data pipelines running to get data into shape for training and using in models. The other half is spent doing tech support for the bunch of recently hired "AI scientists" who can barely code, and who spend their days copy/pasting stuff into various chatbot services. Stuff like telling them how to install python packages and u…

I wonder if this is how the OG VR guys felt in 2016.

Re: Ask HN: Machine learning engineers, what do you do at work?

#66

Post by schmookeeg »

Getting my models dunked on by people who can't open MS Outlook more than 3 tries out of 5, however, have a remarkable depth and insight into their chosen domain of expertise. It's rather humbling.

Collaborating with nontechnical people is oddly my favorite part of doing MLE work right now. It wasn't the case when I did basic web/db stuff. They see me as a magician. I see them as voodoo priests and priestesses. When we get something trained up and forecasting that we both like, it's super fulfilling. I think for both sides.

Most of my modeling is healthcare related. I tease insights out of a monstrous data lake of claims, Rx, doctor notes, vital signs, diagnostic imagery, etc. What is also monstrous is how accessible this information is. HIPAA my left foot.

Since you seemed to be asking about the temporal realities, it's about 3 hours of meetings a week, probably another 3 doing task grooming/preparatory stuff, fixing some ETL problem, or doing a one-off query for the business, the rest is swimming around in the data trying to find a slight edge to forecast something that surprised us for a $million or two using our historical snapshots. It's like playing wheres waldo with math. And the waldo scene ends up being about 50TB or so in size. :D

Re: Ask HN: Machine learning engineers, what do you do at work?

#67

Post by ungamedplayer »

daemonologist wrote: post #55
mysteria wrote: post #22

Earlier quoted context omitted.

I thought most ML engineers use their laptops as dumb terminals and just remote into a Linux GPU server.

Yeah, the workday there looks pretty similar though, except that installing pytorch and pillow is usually no problem. Today it was flash-attn I spent the afternoon on.

Isn't this what containers are for. Someone somewhere gets it configured right and then you download and run pre setup container and add your job data ? Or am I looking at the problem wrong?

Re: Ask HN: Machine learning engineers, what do you do at work?

#68

Post by rolisz »

ungamedplayer wrote: post #67
daemonologist wrote: post #55

Earlier quoted context omitted.

Yeah, the workday there looks pretty similar though, except that installing pytorch and pillow is usually no problem. Today it was flash-attn I spent the afternoon on.

Isn't this what containers are for. Someone somewhere gets it configured right and then you download and run pre setup container and add your job data ? Or am I looking at the problem wrong?

But then how do you test out the latest model that came out from who knows where and has the weirdest dependencies and a super obscure command to install?

Re: Ask HN: Machine learning engineers, what do you do at work?

#69

Post by __rito__ »

davedx wrote: post #5

pip install pytorch Environment broken Spend 4 hours fixing python environment pip install Pillow Something something incorrect cpu architecture for your Macbook Spend another 4 hours reinstalling everything from scratch after nuking every single mention of python pip install … oh time to go home!

Just use conda.

Re: Ask HN: Machine learning engineers, what do you do at work?

#70

Post by SoftTalker »

__rito__ wrote: post #69
davedx wrote: post #5

pip install pytorch Environment broken Spend 4 hours fixing python environment pip install Pillow Something something incorrect cpu architecture for your Macbook Spend another 4 hours reinstalling everything from scratch after nuking every single mention of python pip install … oh time to go home!

Just use conda.

Then you get one of my favorites: NVIDIA- has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running.
Post reply on HN