Getting my models dunked on by people who can't open MS Outlook more than 3 tries out of 5, however, have a remarkable depth and insight into their chosen domain of expertise. It's rather humbling. Collaborating with nontechnical people is oddly my favorite part of doing MLE work right now. It wasn't the case when I did basic web/db stuff. They see me as a magician. I see them as voodoo priests and priestesses. When…
Ask HN: Machine learning engineers, what do you do at work?
191–200 of 233 posts
Re: Ask HN: Machine learning engineers, what do you do at work?
#192I've been doing machine learning since the mid 2000s. About half of my time is spent keeping data pipelines running to get data into shape for training and using in models. The other half is spent doing tech support for the bunch of recently hired "AI scientists" who can barely code, and who spend their days copy/pasting stuff into various chatbot services. Stuff like telling them how to install python packages and u…
Re: Ask HN: Machine learning engineers, what do you do at work?
#193Getting my models dunked on by people who can't open MS Outlook more than 3 tries out of 5, however, have a remarkable depth and insight into their chosen domain of expertise. It's rather humbling. Collaborating with nontechnical people is oddly my favorite part of doing MLE work right now. It wasn't the case when I did basic web/db stuff. They see me as a magician. I see them as voodoo priests and priestesses. When…
The Dead Internet Theory says most activity on the internet is by bots [1]. The Dead Privacy Theory says approximately all private data is not private; but rather is accessible on whim by any data scientist, SWE, analyst, or db admin with access to the database. [1] https://en.wikipedia.org/wiki/Dead_Internet_theory
Re: Ask HN: Machine learning engineers, what do you do at work?
#194pip install pytorch Environment broken Spend 4 hours fixing python environment pip install Pillow Something something incorrect cpu architecture for your Macbook Spend another 4 hours reinstalling everything from scratch after nuking every single mention of python pip install … oh time to go home!
Re: Ask HN: Machine learning engineers, what do you do at work?
#195Earlier quoted context omitted.
Two problems intersect: - You can’t have two versions of the same package in the namespace at the same time. - The Python ecosystem is very bad at backwards compatibility This means that you might require one package that requires foo below version 1.2 and another package that requires foo version 2 and above. There is no good solution to the above problem. This problem is amplified when lots of the packages were wri…
Disk space is cheap, so where it's possible to have 2 (or more) venvs, that seems easiest. The problem with venv is that they don't automatically activate. I've been using a very simple wrapper around python to automatically activate venvs so I can just cd into the directory and do python foo.py and have it use the local venv. I threw it online at https://github.com/fragmede/python-wool/
Now your 15 venvs deep, and have over 3000 different package version combinations installed. Your job is to upgrade them right now because of a level 8 CVE
Re: Ask HN: Machine learning engineers, what do you do at work?
#196the users are researchers and have deep technical knowledge of their use case. it is still a challenge to map their needs into design decisions of what they want in the end. thanks to open-source efforts, the model creation is rather straightforward. but everything around making that happen and shaping it like a tool is a ride.
especially love the ever-changing technical stack of "AI" services by major cloud providers rn. it makes mlops nothing more than a demo imho.
Re: Ask HN: Machine learning engineers, what do you do at work?
#197pip install pytorch Environment broken Spend 4 hours fixing python environment pip install Pillow Something something incorrect cpu architecture for your Macbook Spend another 4 hours reinstalling everything from scratch after nuking every single mention of python pip install … oh time to go home!
I count at least a half dozen “just use X” replies to this comment, for at least a half dozen values of X, where X is some wrapper on top of pip or a replacement for pip or some virtual environment or some alternative to a virtual environment etc etc etc. Why is python dependency management so cancerously bad? Why are there so many “solutions” to this problem that seem to be out of date as soon as they exist? Are pyt…
I can only imagine it seemed like an oasis to R which is bottom tier.
So when you combine data scientists, academics, mathematicians, juniors, grifters selling courses…
things like bad package management, horrible developer experience, absolutely no drive in the ecosystem to improve anything beyond the “pinnacle” of wrapping C libraries are all both inevitable and symptoms of a poorly designed ecosystem.
Re: Ask HN: Machine learning engineers, what do you do at work?
#198In my case, there are established processes and designated teams for cleaning & collecting data, but you still do a part of it yourself to provide guidelines. So, even though data is always a perpetual problem, I can shed off most of that boring stuff.
Ah, and of course you're not a real engineer if you don't spend at least 1-2% of your time explaining to other people (surprisingly often to a technical staff, but not ML-oriented) why doing X is a really bad idea. Or, just explaining how ML systems work with ill-fitted metaphors.
Re: Ask HN: Machine learning engineers, what do you do at work?
#199Earlier quoted context omitted.
You could learn how to use Python. Just spend one of those 4 hours actually learning. Imagine just getting into a car and pressing controls until something happened. This wouldn't be allowed to happen in any other industry.
Could you be a bit more specific about what you mean by "You could learn how to use python"? What resources would you recommend to learn how to work around problems the OP has? What basic procedures/resources can you recommend to "learn python"? I work as a software developer alongside my studies and often face the same problems as OP that I would like to avoid. Very grateful for any tips!
To avoid spending hours on fixing broken environments after a single "pip install", I would make it easy to rollback to a known state. For example, recreate virtualenv from a lock requirements file stored in git: `pipenv sync` (or corresponding command using your preferred tool).
Re: Ask HN: Machine learning engineers, what do you do at work?
#200Earlier quoted context omitted.
So… Is it surprising that Engineers in healthcare dont read the actual HIPAA documentation? Use of health data is permitted so long as it’s for payment, treatment or operations. Disclosures and patient consent are not required. There are helpful summaries on the US Department of Health and Humans Services website of the various rules (Security, Privacy & Notification) Source: https://www.hhs.gov/hipaa/for-professiona…
> If it wasn’t then, theoretically the US healthcare industry would grind to a halt considering the number of intermediaries for a single transaction. It just occurred to me that cleaning up our country's data privacy / data ownership mess might have extraordinarily positive second-order effects on our Kafkaesque and criminally expensive healthcare "system". Maybe making it functionally impossible for there to be hun…