Ask HN: Machine learning engineers, what do you do at work? • Hacker News Board

Re: Ask HN: Machine learning engineers, what do you do at work?

#141

Post by otteromkram » Sat, Jun 08, 2024, 2:06 PM UTC

AndrewKemendo wrote: post #38

tedivm wrote: post #30

Earlier quoted context omitted.

It's not about "yucky" so much as specialization and only having a limited time in life to learn everything. Should your reseacher have to manage nvidia drivers and infiniband networking? Should your operations engineer need to understand the math behind transformers? Does your researcher really gain any value from understanding the intricacies of docker layer caching? I've seen what it looks like when a company hire…

My answer is yes to both of those If other peoples work is reliant on yours then you should know how their part of the system transforms your inputs Similarly you should fully understand how all the inputs to your part of the system are generated No matter your coupling pattern, if you have more than 1 person product, knowing at least one level above and below your stack is a baseline expectation This is true with pe…

The parent comment had three examples...

Re: Ask HN: Machine learning engineers, what do you do at work?

#142

Post by jackspawn » Sat, Jun 08, 2024, 2:10 PM UTC

50%+ of my time is spent on backend engineering because the ML is used inside a bigger API.

I take responsibility for the end to end experience of said API, so I will do whatever gives the best value per time spent. This often has nothing to do with the ML models.

Re: Ask HN: Machine learning engineers, what do you do at work?

#143

Post by Xenoamorphous » Sat, Jun 08, 2024, 2:19 PM UTC

I’m a regular software dev but I’ve had to do ML stuff by necessity.

I wonder how “real” ML people deal with the stochastic/gradient results and people’s expectations.

If I do ordinary software work the thing either works or it doesn’t, and if it doesn’t I can explain why and hopefully fix it.

Now with ML I get asked “why did this text classifier not classify this text correctly?” and all I can say is “it was 0.004 points away to meet the threshold”, and “it didn’t meet it because of the particular choice of words or even their order” which seems to leave everyone dissatisfied.

Re: Ask HN: Machine learning engineers, what do you do at work?

#144

Post by nomilk » Sat, Jun 08, 2024, 2:28 PM UTC

schmookeeg wrote: post #66

Getting my models dunked on by people who can't open MS Outlook more than 3 tries out of 5, however, have a remarkable depth and insight into their chosen domain of expertise. It's rather humbling. Collaborating with nontechnical people is oddly my favorite part of doing MLE work right now. It wasn't the case when I did basic web/db stuff. They see me as a magician. I see them as voodoo priests and priestesses. When…

The Dead Internet Theory says most activity on the internet is by bots [1]. The Dead Privacy Theory says approximately all private data is not private; but rather is accessible on whim by any data scientist, SWE, analyst, or db admin with access to the database.

[1] https://en.wikipedia.org/wiki/Dead_Internet_theory

Re: Ask HN: Machine learning engineers, what do you do at work?

#145

Post by burnedout_dc4e3 » Sat, Jun 08, 2024, 2:33 PM UTC

naveen99 wrote: post #63

burnedout_dc4e3 wrote: post #52

I've been doing machine learning since the mid 2000s. About half of my time is spent keeping data pipelines running to get data into shape for training and using in models. The other half is spent doing tech support for the bunch of recently hired "AI scientists" who can barely code, and who spend their days copy/pasting stuff into various chatbot services. Stuff like telling them how to install python packages and u…

You’re living the dream. Why quit ?

I like to feel useful, and like I'm actually contributing to things. I probably didn't express it well in my first post, but the attitude is very much that my current role is obsolete and a relic that's just sticking around until the AI can do everything.

It means I'm marginalized in terms of planning. The company has long term goals that involve making good use of data. Right now, the plan is that "AI" will get us there, with no plan B is it doesn't work. When it inevitably fails to live up to the hype, we're going to have a bunch of clobbered together systems that are expensive to run, rather than something that we can keep iterating on.

It means I'm marginalized in terms of getting resources for projects. There's a lot of good my team could be doing if we had the extra budget for more engineers and computing. Instead that budget is being sent off to AI services, and expensive engineer time is being spent on tech support for people that slapped "LLM" all over their resume.

Re: Ask HN: Machine learning engineers, what do you do at work?

#146

Post by outside1234 » Sat, Jun 08, 2024, 2:33 PM UTC

jollofricepeas wrote: post #129

saulrh wrote: post #75

Earlier quoted context omitted.

> HIPAA my left foot. That was my experience as well - training documentation for fresh college grads (i.e. me) directed new engineers to just... send SQL queries to production to learn. There was a process for gaining permissions, there were audit logs, but the only sign-off you needed was your manager, permission lasted 12 months, and the managers just rubber-stamped everyone. That was ten years ago. Every time I t…

So… Is it surprising that Engineers in healthcare dont read the actual HIPAA documentation? Use of health data is permitted so long as it’s for payment, treatment or operations. Disclosures and patient consent are not required. There are helpful summaries on the US Department of Health and Humans Services website of the various rules (Security, Privacy & Notification) Source: https://www.hhs.gov/hipaa/for-professiona…

Define operations, because that sounds like a loophole that basically allows you to use it for anything

Re: Ask HN: Machine learning engineers, what do you do at work?

#147

Post by hkt » Sat, Jun 08, 2024, 2:36 PM UTC

Xenoamorphous wrote: post #143

I’m a regular software dev but I’ve had to do ML stuff by necessity. I wonder how “real” ML people deal with the stochastic/gradient results and people’s expectations. If I do ordinary software work the thing either works or it doesn’t, and if it doesn’t I can explain why and hopefully fix it. Now with ML I get asked “why did this text classifier not classify this text correctly?” and all I can say is “it was 0.004 p…

This seems to be the absolute worst of all worlds: the burden of software engineering with the tools of an English Language undergrad.

Re: Ask HN: Machine learning engineers, what do you do at work?

#148

Post by hkt » Sat, Jun 08, 2024, 2:39 PM UTC

llm_trw wrote: post #126

posix_monad wrote: post #123

Earlier quoted context omitted.

Python's dominance is holding us back. We need a stack with a more principled approach to environments and native dependencies.

Here's what getting PyTorch built reproducibly looks like: https://hpc.guix.info/blog/2021/09/whats-in-a-package/ Since then the whole python ecosystem has gotten worse. We are building towers on quicksand. It's not about python, it's about people who don't care about dependencies.

Dependency management is just.. hard. It is one of the things where everything relies upon it but nobody thinks "hey, this is my responsibility to improve" so it is left to people who have the most motivation, academic posts, or grant funding. This is roughly the same problem that led to heartbleed for OpenSSL.

Re: Ask HN: Machine learning engineers, what do you do at work?

#149

Post by avmich » Sat, Jun 08, 2024, 2:41 PM UTC

makapuf wrote: post #104

avmich wrote: post #10

Earlier quoted context omitted.

Legends say there were times when you'd have a program.c file and just run cc program.c, and then could just execute the compiled result. Funny that programmer's job is highly automatable, yet we invent ourselves tons of intermediate layers which we absolutely have to deal with manually.

I agree simplicity is king. But you're comparing making a script using dependencies and tooling for those dependencies and a C program with no dependencies. You can download a simple python script and run it directly if it has no dependencies besides stdlib (which is way larger in python). That's why I love using bottle.py by example.

Agree. But even with dependencies running "make" seems to be way simpler than having to install particular version of tools for a project, making venv and then picking versions of dependencies.

The point is the same - we had it simpler and now, with all capabilities for automation, we have it more complex.

Frankly, I suspect most of the efforts now are spent fighting non-essential complexities, like compatibilities, instead of solving the problem at hand. That means we create problems for ourselves faster than removing them.

Re: Ask HN: Machine learning engineers, what do you do at work?

#150

Post by hkt » Sat, Jun 08, 2024, 2:41 PM UTC

nicce wrote: post #23

sigmoid10 wrote: post #9

Earlier quoted context omitted.

If you're still doing ML locally in 2024 and also use an ARM macbook, you're asking for trouble.

> ARM macbook Funnily, the only real competitor for Nvidias' GPUs are Macbooks with 128GB of RAM.

I see your contemporary hardware choices and raise you my P900 ThinkStation with 256GB of RAM and 48 Xeon cores. Eventually it might even acquire modern graphics hardware.