The opposite of what you’d think when studying machine learning… 95% of the job is data cleaning, joining datasets together and feature engineering. 5% is fitting and testing models.
Sounds like a Data Scientist job?
A good DS can double as an MLE.
Ask HN: Machine learning engineers, what do you do at work?
41–50 of 233 posts
Re: Ask HN: Machine learning engineers, what do you do at work?
#42Earlier quoted context omitted.
And then you'd have to deal with wrong glibc versions or mysterious segfaults or undefined behavior or the the code assuming the wrong arch or ...
python solves none of those issues. It just adds a myriad of ways those problems can get to you. All of a sudden you have people with C problems, who have no idea they're even using compiled dependencies.
In theory you're right, CPython is written in C and it could segfault or display undefined behavior. In practice, you're quite wrong.
It's not really much of a counterargument to say that Python is good enough that you don't have to care what's under the hood, except when it breaks because C sucks so badly.
Re: Ask HN: Machine learning engineers, what do you do at work?
#43Earlier quoted context omitted.
I had a bad experience with Conda: - If they're so good at dependency management, why is Conda installed through a magical shell script? - It's slow as molasses. - Choosing between Anaconda/Miniconda... When forced to use Python, I prefer Poetry, or just pip with freezing the dependencies. The Python people probably can't even imagine how great dependency management is in all the other languages...
Mamba/micromamba solves the slowness problem of conda
To add. Conda has parallelized downloads and is faster. Not as fast as mamba, but faster than previously. pr merged sep 2022 -> https://github.com/conda/conda/pull/11841
Re: Ask HN: Machine learning engineers, what do you do at work?
#44Earlier quoted context omitted.
This is a large problem in industry: defining away some of the most important parts of a job or role as (should be) someone else's. There is a lot of toil and unnecessary toil in the whole data field, but if you define away all of the "yucky" parts, you might find that all of those "someone elses" will end up eating your lunch.
It's not about "yucky" so much as specialization and only having a limited time in life to learn everything. Should your reseacher have to manage nvidia drivers and infiniband networking? Should your operations engineer need to understand the math behind transformers? Does your researcher really gain any value from understanding the intricacies of docker layer caching? I've seen what it looks like when a company hire…
> I've seen what it looks like when a company hires mostly researchers and ignores other expertise, versus what happens when a company hires diverse talent sets to build a cross domain team. The second option works way better.
I've seen these too, and you aren't wrong. Division into specializations can work "way better" (i.e. the overall potential is higher), but in practice the differentiating factors that matter will come down to organizational and ultimately human-factors. The anecdotal cases I draw my observations from organizations operating at the scale of 1-10 people, as well as 1,000s working in this field.
> Should your reseacher have to manage nvidia drivers and infiniband networking? Should your operations engineer need to understand the math behind transformers? Does your researcher really gain any value from understanding the intricacies of docker layer caching?
To realize the higher potential mentioned above, what they need to be doing is appreciating the value of what those things are and those who do those things beyond: these are the people that do the things I don't want to do or don't want to understand. That appreciation usually comes from having done and understanding that work.
When specializations are used, they tend to also manifest into organizational structures and dynamics which are ultimately comprised of humans. Conway's Law is worth mentioning here because the interfaces between these specializations become the bottleneck of your system in realizing that "higher potential."
As another commenter mentions, the effectiveness of these interfaces, corresponding bottlenecking effects, and ultimately the entire people-driven system is very much driven by how the parties on each side understand each other's work/methods/priorities/needs/constraints/etc, and having an appreciation for how they affect (i.e. complement) each other and the larger system.
Re: Ask HN: Machine learning engineers, what do you do at work?
#45Earlier quoted context omitted.
python solves none of those issues. It just adds a myriad of ways those problems can get to you. All of a sudden you have people with C problems, who have no idea they're even using compiled dependencies.
In theory you're right, CPython is written in C and it could segfault or display undefined behavior. In practice, you're quite wrong. It's not really much of a counterargument to say that Python is good enough that you don't have to care what's under the hood, except when it breaks because C sucks so badly.
I was specifically talking about python packages using C. You type "pip install" and god knows what's going to happen. It might pull a precompiled wheel, it might just compile and link some C or Fortran code, it might need external dependecies. It might install flawlessly and crash as soon as you try to run it. All bets are off.
I never experienced CPython itself segfault, it's always due to some package.
Re: Ask HN: Machine learning engineers, what do you do at work?
#46Although I studied machine learning and was originally hired for that role, the company pivoted and is now working with LLMs, so I spend most of my day working on figuring out how different LLMs work, what parameters work best for them, how to do RAG, how to integrate them with other bots.
Would you not consider LLMs as a part of machine learning?
Probably it's because we are not training them anymore and just using with prompts. Seems like more of a swe regular type of job
Re: Ask HN: Machine learning engineers, what do you do at work?
#47pip install pytorch Environment broken Spend 4 hours fixing python environment pip install Pillow Something something incorrect cpu architecture for your Macbook Spend another 4 hours reinstalling everything from scratch after nuking every single mention of python pip install … oh time to go home!
If you're still doing ML locally in 2024 and also use an ARM macbook, you're asking for trouble.
I wish my company would understand this and let us use something else. Luckily, they don't really seem to care that I use my Linux based gaming machine most of the time
Re: Ask HN: Machine learning engineers, what do you do at work?
#48pip install pytorch Environment broken Spend 4 hours fixing python environment pip install Pillow Something something incorrect cpu architecture for your Macbook Spend another 4 hours reinstalling everything from scratch after nuking every single mention of python pip install … oh time to go home!
As an amateur game engine developer, I morosely reflect my hobby seems to actually consist of endlessly chasing things that were broken by environment updates (OS, libraries, compiler, etc.) That is, most of the time I sit down to code I actually spend nuking and reinstalling things that (I thought) were previously working.
Your comment makes me feel a little better that this is not merely some personal failing of focus, but happens in a professional setting too.
Re: Ask HN: Machine learning engineers, what do you do at work?
#49My job title is ML Engineer, but my day to day job is almost pure software engineering. I build the systems to support ML systems in production. As others have mentioned, this includes mostly data transformation, model training, and model serving. Our job is also to support scientists to do their job, either by building tools or modifying existing systems. However, looking outside, I think my company is an outlier. I…
My company is largely the same. I’m an MLE and partner with data scientists. I don’t train or validate the models. I productionize and instrument the feature engineering pipelines and model deployments. More data engineering and MLOps than anything. I’m in a highly regulated industry so the data scientists have many compliance tasks related to the models and we engineers have our own compliance tasks related to the deployments. I was an MLE at another company in the very same industry before and did everything in the model lifecycle and it was just too much.
Re: Ask HN: Machine learning engineers, what do you do at work?
#50I work on optimizing our inference code, "productizing" our trained models and currently I'm working on local training and inference since I work in an industry where cloud services just aren't very commonly used yet. It's super interesting too since it's not LLMs, meaning that there aren't as many pre made tools and we have to make tons of stuff by ourselves. That means touching anything from assessing data quality (again, the local part is the challenge) to using CUDA directly as we already have signal processing libs that are built around it and that we can leverage.
Sometimes it also involves building internal tooling for our team (we are a mixed team of researchers/MLEs), to visualize the data and the inferences as again, it's a pretty niche sector and that means having to build that ourselves. That allowed me to have a lot of impact in my org as we basically have complete freedom w.r.t tooling and internal software design, and one of the tools that I built basically on a whim is now on the way to be shipped in our main products too.