I'm the ML guy at Nyckel. Great to see all thoughts and comments! Feel free to AMA.
Machine learning is still too hard for software engineers
91–100 of 154 posts
Re: Machine learning is still too hard for software engineers
#92I'm the ML guy at Nyckel. Great to see all thoughts and comments! Feel free to AMA.
Can you give a succinct summary of what makes you guys different from datarobot etc?
In general, I feel like ML platforms have the problem outlined by https://xkcd.com/927/ (Standards)
Re: Machine learning is still too hard for software engineers
#93"Machine Learning is too difficult" ... says company selling "ML platform [which] can be used by anyone and it only takes minutes to train your first model." No, actually, you're just being dishonest. Even if you hide TensorFlow and the keras models behind a nice GUI, people still need that mathematics knowledge to succeed. And yes, pre-training is great. But you need a shitload of stochastic analysis to make sure th…
Thanks for your comments fxtentacle!
I work at Nyckel. In fact, I'm the "ml guy" at Nyckel. I have a PhD in ML and did some research at Berkeley, but I mostly consider myself a ML engineer. My most recent job was in the self-driving car industry, leading a ML team there.
Knowing the math/stats is helpful when navigating the vast set of models to choose from when fitting your data. Although I'd argue that some sort of black-magic "intuition" earned by doing this for a long time is more important in practice...
However, when validating a model, there is really only one way: test it on production data. This is what Nyckel does: upload your production data, do some annotations, and see if it works. Nyckel handles model search, cross validation, etc for you which reduces the risk of bugs. In a way we are making the argument that by focusing on your data, you are most likely to do well.
But what about that pesky out-of-domain issue? Like the tank/cats or whatever? Well, our customers are not trying to develop AGI, but solve narrow problems using image and text classification. And they are also doing it for themselves so they have all the incentives to be honest. Consider one example use-case from a health food store we work with: "what type of legume (from the 10 I offer in bulk) is in this picture"? As long as they train and test on production data from the warehouse camera stream, they are in good shape from a statistical perspective. Sure, if they throw in a picture from anywhere else, they are toast, but why would they?
Re: Machine learning is still too hard for software engineers
#94I'm the ML guy at Nyckel. Great to see all thoughts and comments! Feel free to AMA.
Can you give a succinct summary of what makes you guys different from datarobot etc? In general, I feel like ML platforms have the problem outlined by https://xkcd.com/927/ (Standards)
Good question. We have been doing this for almost 2 years now and we still find new players almost every week! It's a bit of a wild west for sure.
I can't say what we do different from everyone, but a few things that we focus on: * Speed: we train models based on DL in seconds. So you get real-time feedback on your model/data as you annotate and upload more. This is true for a few, but far from all of our competitors. In our benchmarking we find that we still perform on par with the competition (at least in the "low-data" regime https://www.nyckel.com/blog/automl-benchmark-nyckel-google-h...) * Level of abstraction: Many competitors expose some ML knobs for their users thinking it will improve the experience. We found that this induces "ML anxiety" for many. As a result we have zero knobs. Just focus on your data, we do the rest. * API: we have spend a ton of time developing clean API abstractions. Some competitors have great APIs, other don't. * Cost: we are super cheap. Our lowest tier if $50. We don't charge for training or per function/model.
Re: Machine learning is still too hard for software engineers
#95So, I went directly into data science after an econ degree, worked there for 2 years and then transitioned to SWE (at startups). First, I am 100% certain ML will become a part of the standard SWE toolkit (just like apis, docker, sql, etc..). However, to the relative "hardness" I would say ML currently is much less things but they can be really hard to get your head around (like starting to think in embeddings and vec…
Standard ML applications will be commoditized in the same way that creating a website has been commoditized. But for anything that isn't standard, and that's the vast majority of what businesses need, ML isn't going to become part of SWE's standard toolkit anytime soon.
ML is kind of like tennis in the sense that you look at Nadal and Federer and all the greats and you can say, "man, I could do that," but you can't, not even close in this life and the next after this one.
In fact, most SWEs who have developed some sort of "intuition" about ML have been quite dangerous, as they tend to be condescending to the real experts, have built things that make no sense at all or fall apart when 1 data point our of 100,000 changes, and when presented with the fact that they have no clue about ML, resort to the "the AI is all hype anyway" comeback. And vice versa (ML practitioners like me who think they can do SWE with one eye closed, "what's the big deal?") is also true.
Re: Machine learning is still too hard for software engineers
#96Earlier quoted context omitted.
Any good tech lead will remove that need, so no, still not good general advice.
Regardless of how low you get your communication overhead, it still exists. It's rare to find people who can both run and test all the infrastructure and model code, and notice that the transformation you apply on line 34876 of file foobar_now_with_added_ml.py is statistically inappropriate for your problem. That's not even to mention the really hard part, selecting a good outcome variable and appropriate ways to mea…
> It's rare to find people who can both run and test all the infrastructure and model code
It's also unnecessary to do so as long as your institutional processes are capable of synthesizing multiple peoples' competencies across multiple disciplines.
How do you think any machine more complicated than a train car was designed? Do my mechanical engineers need to understand the intricacies of avionics?
> it's really incredible to find people who can do both
Absolutely, and I think you'd have a hard time finding someone who disagrees. But you made a very strong assertion about a "need" which requires much stronger arguments to support.
Re: Machine learning is still too hard for software engineers
#97Earlier quoted context omitted.
In my experience, both are true. I'm more on the ML side, and I can tell I don't have the kind of routine and habits that good software engineers have, though I'm learning. But on the other hand, and I've seen this from software engineers who've made the transition to ML, and clearly have a good handle on the concepts (in one case even published papers in ML journals), they don't seem to have the intuition that allow…
I think the above commenter is more critiquing the notion of relying on intuition all together.
Intuition is always important, but it shouldn't be the last word in an engineering problem. I think there is room for a lot more rigour in how we build, optimize, and validate ML model performance, so less of it gets left to intuition. The discipline is becoming mature enough that this is possible, I think there is a lot of room to build out "standards" and a "body of knowledge" that can be applied to building ML models. We're seeing it in pockets, but in so many cases, it is still a dark art.
And then from an actual software engineering perspective, so much ML code is just run-once jupyter notebook stuff... there is a lot we can do. I need to give this more thought, but I think it's acknowledged there is a big opportunity here
Re: Machine learning is still too hard for software engineers
#98I still remember when SQL was a dark science that could only be managed by administrators that were initiated in secret rooms. Nowadays a lot of people use SQL without deep understanding. They do useful stuff but if you are very skilled you can do way more. And SQL experts are often frustrated with all the amateurs that use their database in such a sub-optimal way. I expect the same for ML. The tooling will improve u…
As ML expert, I am often frustrated at the sub-optimal things I find in the wild or even publications. But you need to understand everyone is on a growing trajectory.
Re: Machine learning is still too hard for software engineers
#99Earlier quoted context omitted.
The "deep learning is a black box" meme is about 5 years past it's due date. It's not as tight as for convex models but we do understand what's going on inside, just not perfectly yet.
I think we're talking about different levels of understanding. For things like convex optimization, we have optimality results. For deep learning we have "try to stop training early because it tends to get overfit if you run it too long", "increase the number of parameters in the transformer to magically get uncanny impressions of written text out". These are not the same kind of understanding. If I hand you a 175B p…
Is it actually tensor products? I thought it was some multilinear map that factors through the tensor product, but my knowledge is limited.
Re: Machine learning is still too hard for software engineers
#100I'd argue software engineering is still too hard for ML engineers. Most of ML (not research but commercial applications of what's already been demonstrated) is now well within the realm of engineering, but there are few standard practices, bodies of knowledge, or agreed on processes for doing anything. These are problems that engineering formalism solve, not another auto-ml tool. Maybe I'm saying the same thing as th…
To add to your point, the Hidden Technical Debt in Machine Learning Systems paper: https://proceedings.neurips.cc/paper/2015/file/86df7dcfd896f...
[deleted]