Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Machine learning engineers, what do you do at work?

news.ycombinator.com

Post reply on HN
Print view View original

121–130 of 233 posts

Re: Ask HN: Machine learning engineers, what do you do at work?

#121

Post by Havoc »

bentt wrote: post #65
burnedout_dc4e3 wrote: post #52

I've been doing machine learning since the mid 2000s. About half of my time is spent keeping data pipelines running to get data into shape for training and using in models. The other half is spent doing tech support for the bunch of recently hired "AI scientists" who can barely code, and who spend their days copy/pasting stuff into various chatbot services. Stuff like telling them how to install python packages and u…

I wonder if this is how the OG VR guys felt in 2016.

Well Palmer Luckey sold oculus and now makes military gear so I guess he chose violence after his VR era

Re: Ask HN: Machine learning engineers, what do you do at work?

#122

Post by jamil7 »

toephu2 wrote: post #3
tambourineman88 wrote: post #2

The opposite of what you’d think when studying machine learning… 95% of the job is data cleaning, joining datasets together and feature engineering. 5% is fitting and testing models.

Sounds like a Data Scientist job?

My partner is a data engineer, from what I’ve gathered the departments are often very small or one person so the roles end up blending together a lot.

Re: Ask HN: Machine learning engineers, what do you do at work?

#123

Post by posix_monad »

davedx wrote: post #5

pip install pytorch Environment broken Spend 4 hours fixing python environment pip install Pillow Something something incorrect cpu architecture for your Macbook Spend another 4 hours reinstalling everything from scratch after nuking every single mention of python pip install … oh time to go home!

Python's dominance is holding us back. We need a stack with a more principled approach to environments and native dependencies.

Re: Ask HN: Machine learning engineers, what do you do at work?

#124

Post by ninkendo »

davedx wrote: post #5

pip install pytorch Environment broken Spend 4 hours fixing python environment pip install Pillow Something something incorrect cpu architecture for your Macbook Spend another 4 hours reinstalling everything from scratch after nuking every single mention of python pip install … oh time to go home!

I count at least a half dozen “just use X” replies to this comment, for at least a half dozen values of X, where X is some wrapper on top of pip or a replacement for pip or some virtual environment or some alternative to a virtual environment etc etc etc.

Why is python dependency management so cancerously bad? Why are there so many “solutions” to this problem that seem to be out of date as soon as they exist?

Are python engineers just bad, or?

(Background: I never used python except for one time when I took a coursera ML course and was immediately assaulted with conda/miniconda/venv/pip/etc etc and immediately came away with a terrible impression of the ecosystem.)

Re: Ask HN: Machine learning engineers, what do you do at work?

#125

Post by llm_trw »

davedx wrote: post #5

pip install pytorch Environment broken Spend 4 hours fixing python environment pip install Pillow Something something incorrect cpu architecture for your Macbook Spend another 4 hours reinstalling everything from scratch after nuking every single mention of python pip install … oh time to go home!

Just use Linux. Then you only have to fight the nvidia drivers.

Re: Ask HN: Machine learning engineers, what do you do at work?

#126

Post by llm_trw »

posix_monad wrote: post #123
davedx wrote: post #5

pip install pytorch Environment broken Spend 4 hours fixing python environment pip install Pillow Something something incorrect cpu architecture for your Macbook Spend another 4 hours reinstalling everything from scratch after nuking every single mention of python pip install … oh time to go home!

Python's dominance is holding us back. We need a stack with a more principled approach to environments and native dependencies.

Here's what getting PyTorch built reproducibly looks like: https://hpc.guix.info/blog/2021/09/whats-in-a-package/

Since then the whole python ecosystem has gotten worse.

We are building towers on quicksand.

It's not about python, it's about people who don't care about dependencies.

Re: Ask HN: Machine learning engineers, what do you do at work?

#127

Post by hu3 »

nicce wrote: post #23
sigmoid10 wrote: post #9

Earlier quoted context omitted.

If you're still doing ML locally in 2024 and also use an ARM macbook, you're asking for trouble.

> ARM macbook Funnily, the only real competitor for Nvidias' GPUs are Macbooks with 128GB of RAM.

And they don't compete in performance.

Re: Ask HN: Machine learning engineers, what do you do at work?

#128

Post by navbaker »

SJC_Hacker wrote: post #62
davedx wrote: post #5

pip install pytorch Environment broken Spend 4 hours fixing python environment pip install Pillow Something something incorrect cpu architecture for your Macbook Spend another 4 hours reinstalling everything from scratch after nuking every single mention of python pip install … oh time to go home!

Docker is your friend or pyenv at least

Yes, I’ve switched from conda to a combination of dev containers and pyenv/pyenv-virtualenv on both my Linux and MacBook machines and couldn’t be happier

Re: Ask HN: Machine learning engineers, what do you do at work?

#129

Post by jollofricepeas »

saulrh wrote: post #75
schmookeeg wrote: post #66

Getting my models dunked on by people who can't open MS Outlook more than 3 tries out of 5, however, have a remarkable depth and insight into their chosen domain of expertise. It's rather humbling. Collaborating with nontechnical people is oddly my favorite part of doing MLE work right now. It wasn't the case when I did basic web/db stuff. They see me as a magician. I see them as voodoo priests and priestesses. When…

> HIPAA my left foot. That was my experience as well - training documentation for fresh college grads (i.e. me) directed new engineers to just... send SQL queries to production to learn. There was a process for gaining permissions, there were audit logs, but the only sign-off you needed was your manager, permission lasted 12 months, and the managers just rubber-stamped everyone. That was ten years ago. Every time I t…

So…

Is it surprising that Engineers in healthcare dont read the actual HIPAA documentation?

Use of health data is permitted so long as it’s for payment, treatment or operations. Disclosures and patient consent are not required.

There are helpful summaries on the US Department of Health and Humans Services website of the various rules (Security, Privacy & Notification)

Source: https://www.hhs.gov/hipaa/for-professionals/privacy/guidance...

This allowance is permitted to covered entities and by extension their vendors (business associates) by HIPAA.

If it wasn’t then, theoretically the US healthcare industry would grind to a halt considering the number of intermediaries for a single transaction.

Example:

Doctor writes script -> EHR -> Pharmacy -> Switch -> Clearinghouse —> PA Processing -> PBM/Plan makes determination

Along this flow there are other possible branches and vendors.

It’s beyond complex.

Re: Ask HN: Machine learning engineers, what do you do at work?

#130

Post by sjducb »

ninkendo wrote: post #124
davedx wrote: post #5

pip install pytorch Environment broken Spend 4 hours fixing python environment pip install Pillow Something something incorrect cpu architecture for your Macbook Spend another 4 hours reinstalling everything from scratch after nuking every single mention of python pip install … oh time to go home!

I count at least a half dozen “just use X” replies to this comment, for at least a half dozen values of X, where X is some wrapper on top of pip or a replacement for pip or some virtual environment or some alternative to a virtual environment etc etc etc. Why is python dependency management so cancerously bad? Why are there so many “solutions” to this problem that seem to be out of date as soon as they exist? Are pyt…

Two problems intersect:

- You can’t have two versions of the same package in the namespace at the same time. - The Python ecosystem is very bad at backwards compatibility

This means that you might require one package that requires foo below version 1.2 and another package that requires foo version 2 and above.

There is no good solution to the above problem.

This problem is amplified when lots of the packages were written by academics 10 years ago and are no longer maintained.

The bad solutions are: 1) Have 2 venvs - not always possible and if you keep making venvs you’ll have loads of them. 2) Rewrite your code to only use one library 3) Update one of the libraries 4) Don’t care about the mismatch and cross your fingers that the old one will work with the newer library.

Most of the tooling follows approach 1 or 4

Post reply on HN