Live data from Hacker News

Ask HN: Machine learning engineers, what do you do at work?

news.ycombinator.com

191–200 of 233 posts

Re: Ask HN: Machine learning engineers, what do you do at work?

#191

Getting my models dunked on by people who can't open MS Outlook more than 3 tries out of 5, however, have a remarkable depth and insight into their chosen domain of expertise. It's rather humbling. Collaborating with nontechnical people is oddly my favorite part of doing MLE work right now. It wasn't the case when I did basic web/db stuff. They see me as a magician. I see them as voodoo priests and priestesses. When…

HIPAA your left foot because nobody reads fine print anymore and signs their soul away for a one-time $60 discount.

Re: Ask HN: Machine learning engineers, what do you do at work?

#192

I've been doing machine learning since the mid 2000s. About half of my time is spent keeping data pipelines running to get data into shape for training and using in models. The other half is spent doing tech support for the bunch of recently hired "AI scientists" who can barely code, and who spend their days copy/pasting stuff into various chatbot services. Stuff like telling them how to install python packages and u…

Why help them? Tell your boss you're pointing them to the other ones you helped previously.

Re: Ask HN: Machine learning engineers, what do you do at work?

#193
post #144

Getting my models dunked on by people who can't open MS Outlook more than 3 tries out of 5, however, have a remarkable depth and insight into their chosen domain of expertise. It's rather humbling. Collaborating with nontechnical people is oddly my favorite part of doing MLE work right now. It wasn't the case when I did basic web/db stuff. They see me as a magician. I see them as voodoo priests and priestesses. When…

The Dead Internet Theory says most activity on the internet is by bots [1]. The Dead Privacy Theory says approximately all private data is not private; but rather is accessible on whim by any data scientist, SWE, analyst, or db admin with access to the database. [1] https://en.wikipedia.org/wiki/Dead_Internet_theory

At least more of that access is logged now. It used to be only the production databases that were properly logged, now it’s more common for every query to be logged, even in dev environments. The next step will be more monitoring of those logs.

Re: Ask HN: Machine learning engineers, what do you do at work?

#194
post #5

pip install pytorch Environment broken Spend 4 hours fixing python environment pip install Pillow Something something incorrect cpu architecture for your Macbook Spend another 4 hours reinstalling everything from scratch after nuking every single mention of python pip install … oh time to go home!

Really can't recommend Nix for Python stuff more, it's by far the best at managing dependencies that use FFI/native extensions. It can be a pain sometimes to port your existing Poetry/etc. project to work with nix2lang converters but really pays off.

Re: Ask HN: Machine learning engineers, what do you do at work?

#195
post #130

Earlier quoted context omitted.

Two problems intersect: - You can’t have two versions of the same package in the namespace at the same time. - The Python ecosystem is very bad at backwards compatibility This means that you might require one package that requires foo below version 1.2 and another package that requires foo version 2 and above. There is no good solution to the above problem. This problem is amplified when lots of the packages were wri…

Disk space is cheap, so where it's possible to have 2 (or more) venvs, that seems easiest. The problem with venv is that they don't automatically activate. I've been using a very simple wrapper around python to automatically activate venvs so I can just cd into the directory and do python foo.py and have it use the local venv. I threw it online at https://github.com/fragmede/python-wool/

You’re already managing a few hundred dependencies and their versions. Each venv roughly doubles the number of dependencies and they all have slightly different versions.

Now your 15 venvs deep, and have over 3000 different package version combinations installed. Your job is to upgrade them right now because of a level 8 CVE

Re: Ask HN: Machine learning engineers, what do you do at work?

#196
junior level role, but currently it is a mix of working like a proxy product owner and half software engg.

the users are researchers and have deep technical knowledge of their use case. it is still a challenge to map their needs into design decisions of what they want in the end. thanks to open-source efforts, the model creation is rather straightforward. but everything around making that happen and shaping it like a tool is a ride.

especially love the ever-changing technical stack of "AI" services by major cloud providers rn. it makes mlops nothing more than a demo imho.

Re: Ask HN: Machine learning engineers, what do you do at work?

#197
post #5

pip install pytorch Environment broken Spend 4 hours fixing python environment pip install Pillow Something something incorrect cpu architecture for your Macbook Spend another 4 hours reinstalling everything from scratch after nuking every single mention of python pip install … oh time to go home!

I count at least a half dozen “just use X” replies to this comment, for at least a half dozen values of X, where X is some wrapper on top of pip or a replacement for pip or some virtual environment or some alternative to a virtual environment etc etc etc. Why is python dependency management so cancerously bad? Why are there so many “solutions” to this problem that seem to be out of date as soon as they exist? Are pyt…

Unironically yes, it really is that bad. A moderately bad language that happened to have some popular data science and math libraries from the beginning.

I can only imagine it seemed like an oasis to R which is bottom tier.

So when you combine data scientists, academics, mathematicians, juniors, grifters selling courses…

things like bad package management, horrible developer experience, absolutely no drive in the ecosystem to improve anything beyond the “pinnacle” of wrapping C libraries are all both inevitable and symptoms of a poorly designed ecosystem.

Re: Ask HN: Machine learning engineers, what do you do at work?

#198
In larger companies, and, specifically, bigger projects, systems tend to have multiple ML components, and those are usually a mix of large NN models and more classical (ML) algorithms, so you end up tweaking multiple parts at once. In my case optimising for such systems is ~90% of the work. For instance, can I make the model lighter or go faster and keep the performance? Or, can I make it go faster? Loss change, pruning, quantisation, dataset optimisation etc. -- most of the time I'm testing out those options & tweaking parameters. There is of course the deployment part, but this one is usually a quickie if your team has specific processes/pipelines for this. There's a checklist of what you must do while deploying, along with cost targets.

In my case, there are established processes and designated teams for cleaning & collecting data, but you still do a part of it yourself to provide guidelines. So, even though data is always a perpetual problem, I can shed off most of that boring stuff.

Ah, and of course you're not a real engineer if you don't spend at least 1-2% of your time explaining to other people (surprisingly often to a technical staff, but not ML-oriented) why doing X is a really bad idea. Or, just explaining how ML systems work with ill-fitted metaphors.

Re: Ask HN: Machine learning engineers, what do you do at work?

#199

Earlier quoted context omitted.

You could learn how to use Python. Just spend one of those 4 hours actually learning. Imagine just getting into a car and pressing controls until something happened. This wouldn't be allowed to happen in any other industry.

Could you be a bit more specific about what you mean by "You could learn how to use python"? What resources would you recommend to learn how to work around problems the OP has? What basic procedures/resources can you recommend to "learn python"? I work as a software developer alongside my studies and often face the same problems as OP that I would like to avoid. Very grateful for any tips!

How do you learn anything in the space of software engineering? In general, there are many different problems and even more solutions with different tradeoffs.

To avoid spending hours on fixing broken environments after a single "pip install", I would make it easy to rollback to a known state. For example, recreate virtualenv from a lock requirements file stored in git: `pipenv sync` (or corresponding command using your preferred tool).

Re: Ask HN: Machine learning engineers, what do you do at work?

#200

Earlier quoted context omitted.

So… Is it surprising that Engineers in healthcare dont read the actual HIPAA documentation? Use of health data is permitted so long as it’s for payment, treatment or operations. Disclosures and patient consent are not required. There are helpful summaries on the US Department of Health and Humans Services website of the various rules (Security, Privacy & Notification) Source: https://www.hhs.gov/hipaa/for-professiona…

> If it wasn’t then, theoretically the US healthcare industry would grind to a halt considering the number of intermediaries for a single transaction. It just occurred to me that cleaning up our country's data privacy / data ownership mess might have extraordinarily positive second-order effects on our Kafkaesque and criminally expensive healthcare "system". Maybe making it functionally impossible for there to be hun…

Only if your mortgage isn’t paid by such a middleman.
Post reply on HN