31–40 of 233 posts
pip install pytorch Environment broken Spend 4 hours fixing python environment pip install Pillow Something something incorrect cpu architecture for your Macbook Spend another 4 hours reinstalling everything from scratch after nuking every single mention of python pip install … oh time to go home!
Earlier quoted context omitted.
Legends say there were times when you'd have a program.c file and just run cc program.c, and then could just execute the compiled result. Funny that programmer's job is highly automatable, yet we invent ourselves tons of intermediate layers which we absolutely have to deal with manually.
And then you'd have to deal with wrong glibc versions or mysterious segfaults or undefined behavior or the the code assuming the wrong arch or ...
All of a sudden you have people with C problems, who have no idea they're even using compiled dependencies.
Earlier quoted context omitted.
Can recommend using conda, more specifically mambaforge/micromamba (no licensing issues when used at work). This works way better than pip, as it does more checks/dependency checking, so it does not break as easily as pip, though this makes it definitely way slower when installing something. It also supports updating your environment to the newest versions of all packages. It's no silver bullet and mixing it with pip…
I had a bad experience with Conda: - If they're so good at dependency management, why is Conda installed through a magical shell script? - It's slow as molasses. - Choosing between Anaconda/Miniconda... When forced to use Python, I prefer Poetry, or just pip with freezing the dependencies. The Python people probably can't even imagine how great dependency management is in all the other languages...
The opposite of what you’d think when studying machine learning… 95% of the job is data cleaning, joining datasets together and feature engineering. 5% is fitting and testing models.
At the staff/principal level it’s all about maintaining “data impedance” between the product features that rely on inference models and the data capture
This is to ensure that as the product or features change it doesn’t break the instrumentation and data granularity that feed your data stores and training corpus
For RL problems however it’s about making sure you have the right variables captured for state and action space tuple and then finding how to adjust the interfaces or environment models for reward feedback
My job title is ML Engineer, but my day to day job is almost pure software engineering. I build the systems to support ML systems in production. As others have mentioned, this includes mostly data transformation, model training, and model serving. Our job is also to support scientists to do their job, either by building tools or modifying existing systems. However, looking outside, I think my company is an outlier. I…
I gave a talk at the Open Source Summit on MLOps in April, and one of the big points I try to drive home is that it's 80% software development and 20% ML.
Although I studied machine learning and was originally hired for that role, the company pivoted and is now working with LLMs, so I spend most of my day working on figuring out how different LLMs work, what parameters work best for them, how to do RAG, how to integrate them with other bots.
Earlier quoted context omitted.
Can recommend using conda, more specifically mambaforge/micromamba (no licensing issues when used at work). This works way better than pip, as it does more checks/dependency checking, so it does not break as easily as pip, though this makes it definitely way slower when installing something. It also supports updating your environment to the newest versions of all packages. It's no silver bullet and mixing it with pip…
I had a bad experience with Conda: - If they're so good at dependency management, why is Conda installed through a magical shell script? - It's slow as molasses. - Choosing between Anaconda/Miniconda... When forced to use Python, I prefer Poetry, or just pip with freezing the dependencies. The Python people probably can't even imagine how great dependency management is in all the other languages...
Earlier quoted context omitted.
This is a large problem in industry: defining away some of the most important parts of a job or role as (should be) someone else's. There is a lot of toil and unnecessary toil in the whole data field, but if you define away all of the "yucky" parts, you might find that all of those "someone elses" will end up eating your lunch.
It's not about "yucky" so much as specialization and only having a limited time in life to learn everything. Should your reseacher have to manage nvidia drivers and infiniband networking? Should your operations engineer need to understand the math behind transformers? Does your researcher really gain any value from understanding the intricacies of docker layer caching? I've seen what it looks like when a company hire…
If other peoples work is reliant on yours then you should know how their part of the system transforms your inputs
Similarly you should fully understand how all the inputs to your part of the system are generated
No matter your coupling pattern, if you have more than 1 person product, knowing at least one level above and below your stack is a baseline expectation
This is true with personnel leadership too, I should be able to troubleshoot one level above and below me to some level of capacity.
pip install pytorch Environment broken Spend 4 hours fixing python environment pip install Pillow Something something incorrect cpu architecture for your Macbook Spend another 4 hours reinstalling everything from scratch after nuking every single mention of python pip install … oh time to go home!
Although I studied machine learning and was originally hired for that role, the company pivoted and is now working with LLMs, so I spend most of my day working on figuring out how different LLMs work, what parameters work best for them, how to do RAG, how to integrate them with other bots.
Would you not consider LLMs as a part of machine learning?