Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Machine learning engineers, what do you do at work?

news.ycombinator.com

Post reply on HN
Print view View original

181–190 of 233 posts

Re: Ask HN: Machine learning engineers, what do you do at work?

#181

Post by globular-toast »

mountainriver wrote: post #135
globular-toast wrote: post #107

Earlier quoted context omitted.

You could learn how to use Python. Just spend one of those 4 hours actually learning. Imagine just getting into a car and pressing controls until something happened. This wouldn't be allowed to happen in any other industry.

Oh wow, I’ve been a Python engineer for over a decade and getting dependencies right for machine learning has very little to do with Python and everything to do with c++/cuda

I've done it. Isn't it just following instructions? What part of that means destroying every mention of Python on the system?

Re: Ask HN: Machine learning engineers, what do you do at work?

#182

Post by gopher_space »

llama_person wrote: post #137
tambourineman88 wrote: post #2

The opposite of what you’d think when studying machine learning… 95% of the job is data cleaning, joining datasets together and feature engineering. 5% is fitting and testing models.

Same here, it's tons of work to collect, clean, validate data, followed by a tiny fun portion where you train models, then you do the whole loop over again.

> it's tons of work to collect, clean, validate data

That's my fun part. The discovery process is a joy especially if it means ingesting a whole new domain and meeting people.

Re: Ask HN: Machine learning engineers, what do you do at work?

#183

Post by globular-toast »

ninkendo wrote: post #124
davedx wrote: post #5

pip install pytorch Environment broken Spend 4 hours fixing python environment pip install Pillow Something something incorrect cpu architecture for your Macbook Spend another 4 hours reinstalling everything from scratch after nuking every single mention of python pip install … oh time to go home!

I count at least a half dozen “just use X” replies to this comment, for at least a half dozen values of X, where X is some wrapper on top of pip or a replacement for pip or some virtual environment or some alternative to a virtual environment etc etc etc. Why is python dependency management so cancerously bad? Why are there so many “solutions” to this problem that seem to be out of date as soon as they exist? Are pyt…

It's not bad. It works really well. There's always room for improvement. That's technology for you. Python probably does attract more than it's fair share of bad engineers, though.

Re: Ask HN: Machine learning engineers, what do you do at work?

#184

Post by dr_kiszonka »

ProjectArcturis wrote: post #162
schmookeeg wrote: post #66

Getting my models dunked on by people who can't open MS Outlook more than 3 tries out of 5, however, have a remarkable depth and insight into their chosen domain of expertise. It's rather humbling. Collaborating with nontechnical people is oddly my favorite part of doing MLE work right now. It wasn't the case when I did basic web/db stuff. They see me as a magician. I see them as voodoo priests and priestesses. When…

What business are you in that predicting health data can make you millions?

Insurance, health benefits.

Re: Ask HN: Machine learning engineers, what do you do at work?

#185

Post by jononor »

makapuf wrote: post #8
davedx wrote: post #5

pip install pytorch Environment broken Spend 4 hours fixing python environment pip install Pillow Something something incorrect cpu architecture for your Macbook Spend another 4 hours reinstalling everything from scratch after nuking every single mention of python pip install … oh time to go home!

Maybe pip should not work by default (but python -m venv then pip install should)

Some Linux distros are moving that way, particularly for the included Python/pip version. My Arch Linux already does so some years, and I do not set it up myself - so I think it is default.

Re: Ask HN: Machine learning engineers, what do you do at work?

#186

Post by jononor »

eb0la wrote: post #175
shoggouth wrote: post #171

Earlier quoted context omitted.

Could you talk more about “BTW: perl is much better than python on this.”?

I haven't touched Perl in more than 20 years... ... but I (routinely) miss something like: $variable = something() if sanity_check() And do_something() unless $dont_do_that

There exists a ternary if statement?

foo = something() if sanity_check else None

Can replace None with foo (or any other expression), if desired.

Re: Ask HN: Machine learning engineers, what do you do at work?

#187

Post by connicpu »

outside1234 wrote: post #146
jollofricepeas wrote: post #129

Earlier quoted context omitted.

So… Is it surprising that Engineers in healthcare dont read the actual HIPAA documentation? Use of health data is permitted so long as it’s for payment, treatment or operations. Disclosures and patient consent are not required. There are helpful summaries on the US Department of Health and Humans Services website of the various rules (Security, Privacy & Notification) Source: https://www.hhs.gov/hipaa/for-professiona…

Define operations, because that sounds like a loophole that basically allows you to use it for anything

Basically the only thing you can't do with the data is disclose it to someone who doesn't also fall under HIPAA

Re: Ask HN: Machine learning engineers, what do you do at work?

#188

Post by fragmede »

sjducb wrote: post #130
ninkendo wrote: post #124

Earlier quoted context omitted.

I count at least a half dozen “just use X” replies to this comment, for at least a half dozen values of X, where X is some wrapper on top of pip or a replacement for pip or some virtual environment or some alternative to a virtual environment etc etc etc. Why is python dependency management so cancerously bad? Why are there so many “solutions” to this problem that seem to be out of date as soon as they exist? Are pyt…

Two problems intersect: - You can’t have two versions of the same package in the namespace at the same time. - The Python ecosystem is very bad at backwards compatibility This means that you might require one package that requires foo below version 1.2 and another package that requires foo version 2 and above. There is no good solution to the above problem. This problem is amplified when lots of the packages were wri…

Disk space is cheap, so where it's possible to have 2 (or more) venvs, that seems easiest. The problem with venv is that they don't automatically activate. I've been using a very simple wrapper around python to automatically activate venvs so I can just cd into the directory and do python foo.py and have it use the local venv.

I threw it online at https://github.com/fragmede/python-wool/

Re: Ask HN: Machine learning engineers, what do you do at work?

#189

Post by hamasho »

aswegs8 wrote: post #151
hamasho wrote: post #82

Earlier quoted context omitted.

I worked on a project to analyze endoscope videos to find diseases. I examined a lot of images and videos annotated with symptoms of various diseases labeled by assistants and doctors. Most of them are really obvious, but others are almost impossible to detect. In rare cases, despite my best efforts, I couldn't see any difference between the spot labeled as a symptom of cancer and the surrounding area. There's no a-h…

Mind sharing how to get a foot into the field? I've got a good amount of domain knowledge from my studies in life science and rather meager experience from learning to code on my own for a few years. It seems like I cant compete with CS majors and gotta find a way to leverage my domain knowledge.

I'm not an expert in machine learning, but rather a web developer and data engineer helping develop a system to detect diseases from endoscopy images using the model developed by other ML engineers. And it was 5 years ago when I worked on the project, so please take it with a grain of salt.

If you want to learn machine learning for healthcare in general, it may help to start problems with tabular data like CSVs instead of images. Image processing is a lot harder, and takes a lot of time and computational power. But it's best to learn what you're interested in the most.

Anyway, first you need to be familiar with basic; Python, machine learning, and popular libraries like scikit-learn, matplotlib, numpy, and pandas. Those are tons of articles, textbooks, and videos to help you learn them.

If you grasp the basics, I think it's better to learn from actual code to train/evaluate models rather than more theories. Kaggle may be a good starting point. They host a lot of competitions for machine learning problems. There are easy competitions for beginners, and competitions and datasets in the medical field.

You can view notebooks (actual code to solve those problems well written by experts) and popular ones are very educational. You can learn a lot by reading those code, understanding concepts and how to use libraries, and modifying some code to see how it changes the result. ChatGPT is also helpful.

If you want to learn image classification, the technology used to detect objects from images and videos is called image classification and object detection. It uses CNN, one of the deep neural networks. You also need to learn basic image processing, how to train a deep neural network, how to evaluate, and libraries like OpenCV/Pillow/PyTorch/TorchVision. There are a lot of image classification competitions in the medical field on Kaggle too[0][1].

To run those notebooks, I recommend Google Colab. Image processing often uses a lot of GPUs, and you may not have GPUs, or even if you have it's difficult to set up the right environment. It's easier to use those dedicated cloud services and it doesn't cost much.

It's hard to learn, but sometimes fun, so enjoy your journey!

[0] https://www.kaggle.com/datasets/paultimothymooney/chest-xray... [1] https://www.kaggle.com/code/arkapravagupta/endoscopy-multicl...

Re: Ask HN: Machine learning engineers, what do you do at work?

#190

Post by throwaway936r8 »

aiforecastthway wrote: post #136
jollofricepeas wrote: post #129

Earlier quoted context omitted.

So… Is it surprising that Engineers in healthcare dont read the actual HIPAA documentation? Use of health data is permitted so long as it’s for payment, treatment or operations. Disclosures and patient consent are not required. There are helpful summaries on the US Department of Health and Humans Services website of the various rules (Security, Privacy & Notification) Source: https://www.hhs.gov/hipaa/for-professiona…

> If it wasn’t then, theoretically the US healthcare industry would grind to a halt considering the number of intermediaries for a single transaction. It just occurred to me that cleaning up our country's data privacy / data ownership mess might have extraordinarily positive second-order effects on our Kafkaesque and criminally expensive healthcare "system". Maybe making it functionally impossible for there to be hun…

I think it would have the opposite effect. We could end up with a few all-in-one systems that would dominate the market and have little incentive to improve or compete on usability and price.
Post reply on HN