Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Machine learning is still too hard for software engineers

nyckel.com

Post reply on HN
Print view View original

131–140 of 154 posts

Re: Machine learning is still too hard for software engineers

#131

Post by LadyCailin »

zcw100 wrote: post #57
version_five wrote: post #4

I'd argue software engineering is still too hard for ML engineers. Most of ML (not research but commercial applications of what's already been demonstrated) is now well within the realm of engineering, but there are few standard practices, bodies of knowledge, or agreed on processes for doing anything. These are problems that engineering formalism solve, not another auto-ml tool. Maybe I'm saying the same thing as th…

I'd argue software engineering is still too hard for software engineers.

I think industry and society just has too low a bar for software development. If the quality of civil engineers matched that of software engineers as a whole, we’d all be dead from the collapsed buildings and bridges. Of course, software isn’t usually such high stakes, but I don’t think it would hurt the industry to have some formal licensing. If you write a terrible API or whatever, then a review board decides if your license to practice should be removed. The field would suddenly get better, as developers would no longer care about the PMs artificial deadline, if they need more time to write quality code, they will take it, and the PM simply can’t find anyone else to cheap out on the work either.

Now, how you enforce this, I don’t know, but this is my pie in the sky dream.

Re: Machine learning is still too hard for software engineers

#132

Post by ChuckNorris89 »

deepsun wrote: post #123
mrfusion wrote: post #30

Earlier quoted context omitted.

Can you share how you switched into ML? How and what did you study? Any tips?

For English-speakers -- Kaggle forums. Top kaggler (Giba) said he learned everything just reading Kaggle forums (he was an electric engineer). I personally mostly talked to friends, Kaggle, and 1-2 Coursera classes. PS: Kaggle is pretty far from real production ML, but good enough to dive in.

>said he learned everything just reading Kaggle forums (he was an electric engineer)

The math you learn when studying electrical engineering plus all the signal processing stuff, gives you a good foundation to pivot to ML.

It's a shame electrical engineering is so poorly paid by default in comparison to SW dev.

Re: Machine learning is still too hard for software engineers

#133

Post by Xcelerate »

uejfiweun wrote: post #88
Xcelerate wrote: post #58

Coming from a background in computational quantum chemistry, it’s interesting to see all of the people who say ML is “easy” after taking a few online courses and reading some books on data science. If it’s so easy, invent AGI then, since that is the holy grail of machine learning. Most of these people claiming expertise do not have a deep grasp of the mathematical fundamentals required for state of the art research i…

Not to disagree with your point, but your comment really reminded me of the following scene from Breaking Bad: https://www.youtube.com/watch?v=W_dxteeedgs

Haha, I’d never seen that. I suppose my comment is a bit over-the-top compared to my comment history, but it’s somewhat of a touchy point for me when someone jumps into a new field and claims it’s easy and that they’re an expert without being able to back up the claim (some rare people can do this with a new field; most cannot).

Re: Machine learning is still too hard for software engineers

#134

Post by Xcelerate »

JBits wrote: post #114
Xcelerate wrote: post #58

Coming from a background in computational quantum chemistry, it’s interesting to see all of the people who say ML is “easy” after taking a few online courses and reading some books on data science. If it’s so easy, invent AGI then, since that is the holy grail of machine learning. Most of these people claiming expertise do not have a deep grasp of the mathematical fundamentals required for state of the art research i…

What does representation theory and Fourier transforms have to do with neural networks (I've not properly studied ML)?

In the majority of cases, not much. My point is more that many people who claim expertise in machine learning do not have a holistic understanding of the field; their knowledge is patchy and they aren’t able to implement a solution to more than the most general of problems. It would be like a car mechanic that can change the oil but doesn’t know how to locate and fix a leak.

Most neural networks that perform well nowadays are highly specialized to a particular problem domain, and I gave an example of an approach that might be used by someone designing a neural network that is invariant to certain types of symmetries on the input data. This isn’t something a typical DS/ML bootcamper would know how to handle, or even how to approach, despite their claims of expertise.

Re: Machine learning is still too hard for software engineers

#135

Post by ngc248 »

jstx1 wrote: post #39
uoaei wrote: post #27

Earlier quoted context omitted.

In-built is the assumption that one person must have both competencies. This may be true for cash-strapped startups but this hardly plays well as general advice.

It's not an assumption, that's my main point - you need both skills in the same person. And it's true everywhere including in the biggest companies.

Definitely not ... I am a SWE in a big company and working on ML projects for the past 4 years. We have a data scientist team which does the data exploration and comes up with a model. Their output would be a jupyter notebook.

Then we have a team of "Applied ML practitioners", which I am a part of, we productionize the jupyter notebook, by setting up pipelines, services etc. We understand ML algos, stats, probability etc, but not as much as our data scientist team does.

Having both in the same person would be good, but is not necessary.

Re: Machine learning is still too hard for software engineers

#136

Post by kellymore »

I think it goes without saying that this is too relative. One thing that is common in both fields is the level of research and reading docs it entails.

I used to tell my students "software engineering is 70% reading and 30% coding". This remained consistent as I dove into Data Science and now ML with computer vision at Roboflow.

Of course the first time I was exposed to it during a fellowship, I thought I was out of my depths, but this comes with everything new.

To @deepsun point, I've found Kaggle's intro courses quite excellent as well.

Re: Machine learning is still too hard for software engineers

#137

Post by qazpot »

LadyCailin wrote: post #131
zcw100 wrote: post #57

Earlier quoted context omitted.

I'd argue software engineering is still too hard for software engineers.

I think industry and society just has too low a bar for software development. If the quality of civil engineers matched that of software engineers as a whole, we’d all be dead from the collapsed buildings and bridges. Of course, software isn’t usually such high stakes, but I don’t think it would hurt the industry to have some formal licensing. If you write a terrible API or whatever, then a review board decides if yo…

Your suggestion is extremely laughable and naive and reeks of elitism.

The domains where the quality of code has very high stakes do have very strict processes and have an extremely high bar.

The domains where the stakes are not high, for ex - implementing an API to download user reports, have a very low bar.

And your point about cancelling the license to code if you implement a bad API, that's like cancelling a journalist because they made a typo in their article and it went out.

Re: Machine learning is still too hard for software engineers

#138

Post by disgruntledphd2 »

Salgat wrote: post #112
eyegor wrote: post #106

Earlier quoted context omitted.

This is not true at all unless you're only referring to commodity applications. There's no preexisting model for general regression or seq to seq, for instance. You can say, oh, that's a good fit for an LSTM, but you still need to tune the individual layers to suit the application. Although I am slightly conflating ML and DL, since you mention training which heavily implies DL.

Most folks can get away with logistic or linear regression. A tiny portion of those might need to move to random forest. And an even tinier portion may justify deep neural networks. The vast majority of ML is not deep learning, nor is it necessary. We're talking very boring typical business cases.

Yeah, this fixation with deep learning is kinda weird, given that most companies don't have vast amounts of images/text lying around and a pressing business need to understand them.

I'm a big fan of starting simple, which means linear/log regression (normally with lasso, as it does variable selection).

Then, if you can prove business value, it may make sense to start trying to use more unstructured data.

Re: Machine learning is still too hard for software engineers

#139

Post by disgruntledphd2 »

uoaei wrote: post #96
disgruntledphd2 wrote: post #79

Earlier quoted context omitted.

Regardless of how low you get your communication overhead, it still exists. It's rare to find people who can both run and test all the infrastructure and model code, and notice that the transformation you apply on line 34876 of file foobar_now_with_added_ml.py is statistically inappropriate for your problem. That's not even to mention the really hard part, selecting a good outcome variable and appropriate ways to mea…

> It's rare to find people who can both run and test all the infrastructure and model code It's also unnecessary to do so as long as your institutional processes are capable of synthesizing multiple peoples' competencies across multiple disciplines. How do you think any machine more complicated than a train car was designed? Do my mechanical engineers need to understand the intricacies of avionics? > it's really incr…

> It's also unnecessary to do so as long as your institutional processes are capable of synthesizing multiple peoples' competencies across multiple disciplines.

You clearly work at much better run companies than I do ;)

Re: Machine learning is still too hard for software engineers

#140

Post by Eridrus »

deepsun wrote: post #19

I've dived into ML (and DL) with 17 years of software development experience. I'd say it's much easier than software. Yes, there's A TON to learn and experiment with, but still much less than with software. I was able to feel confident enough after just 1.5 years learning and kaggling, and passed easily ML interviews to SF Bay Area companies (hint -- all data science people are extremely glad to see software experien…

Beyond what makes a good model, IME at a FAANG building an ML product, the bulk of the work in practice tends to be general software engineering. There need to be a sufficient number of people who understand the actual ML pieces under the hood, but even when you're making changes to the models, the bulk of the actual work is not complex ideation, but implementing the ideas in software, and this implementation is usually something you can learn in a few months.
Post reply on HN