Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Zen and the Art of Machine Learning Research

blog.jxmo.io

Post reply on HN
Print view View original

11–20 of 112 posts

Re: Zen and the Art of Machine Learning Research

#11

Post by rented_mule »

Around 2015, I found myself managing back end and machine learning engineers (not researchers) at the same time. Many of the back end engineers wanted to do more ML. Some of them did well when given a chance, but others wanted to revert to back end within a few months. At the same time, one of the ML leaders wanted to step away from ML and only do back end work to support ML.

As I studied these dynamics, something occurred to me... Different people need to see signs of success at different frequencies. Because of the nature of our product, measuring the performance of a new/updated model required the model to be live for at least a full calendar month. So, between initial work and final analysis, it was often a 2 month wait or more. For many back end tasks, you can build a quick prototype, run it to see if it works, and be on your way - the signals come all day long. The varying frequency needs of different people went a long way to determining which of them liked working on ML.

This is sort of a manager's version of feature engineering. ;-) The people on that team taught me a lot!

Re: Zen and the Art of Machine Learning Research

#12

Post by 59nadir »

lostdog wrote: post #2

I have some coworkers that are similar in everything--education, work ethic, and intelligence--but some of the tick out ML ideas that work like clockwork, while others get hits rarely if ever. I cannot tell what makes it work for some and not others. Their ideas both sound equally good. Sometimes a coworker will be an ML star for a year or two, but then suddenly run out of steam. It's brutal to watch. I used to think…

Wow, this makes ML sound even more like voodoo than I thought. Can you give examples of what the nature of these ideas is?

Re: Zen and the Art of Machine Learning Research

#13

Post by stared »

It revolves around the sentiment of "go deeper" - but I think it is a double-edged sword. Sure, entropy, tensors and gradients are important - and yes, they are pretty much requirements.

But from what I see, it is the opposite - a lot (if not virtually all) progress in the last decade of deep learning was not because of a fundamental idea, but incremental, experimentally-verified practice. Even though I think there is good intuition for why ReLU is better than sigmoid (tl;dr: last layer is log(sigmoid) ~ ReLU, putting anything different inside kills the gradient), the original paper by Hinton himself was more or less "because it trains 3x faster".

Re-thinking fundamentals might help, but most "let's change the fundamentals" is rarely how it works. Even the most seminal papers, i.e. AlexNet and "Attention Is All You Need", are refinements of existing ideas, and show how they help.

Machine learning is an experimental science. Many mathematically cool ideas do not work. Many engineering ones do.

> I've tweeted before that one of the most important traits in a researcher is healthy paranoia. Be paranoid!

I have seen so many PhDs burned out to cinders; I don't think it is any more a good piece of advice than "depression is good for philosophers". Sure, be a relentless explorer.

> In short, holding on to ideas for too long can actually be counterproductive. Stay open-minded and refuse to let ego cloud your judgement.

Which I think is true.

Re: Zen and the Art of Machine Learning Research

#14

Post by WithinReason »

> If you want to solve a problem, the tried-and-true path to success is to attempt a solution, try it, reach a bottleneck, try to solve it, and only reach for literature when you’ve run out of ideas yourself.

I've found this to be the right balance between using your creativity and getting stuck too long

Re: Zen and the Art of Machine Learning Research

#15

Post by [deleted] »

jdw64 wrote: post #4

I feel that the Zen used in the West and the Zen in East Asia are quite different. I think the Western Zen is probably the one from the 1970s book Zen and the Art of Motorcycle Maintenance. It usually carries a sense of equanimity and beginner's mind. But in East Asia, Zen actually emphasizes aimlessness or non‑purposefulness. The point where I really feel the difference is that Western Zen seems to be about how to t…

[deleted]

Re: Zen and the Art of Machine Learning Research

#16

Post by isoprophlex »

jdw64 wrote: post #4

I feel that the Zen used in the West and the Zen in East Asia are quite different. I think the Western Zen is probably the one from the 1970s book Zen and the Art of Motorcycle Maintenance. It usually carries a sense of equanimity and beginner's mind. But in East Asia, Zen actually emphasizes aimlessness or non‑purposefulness. The point where I really feel the difference is that Western Zen seems to be about how to t…

"To be done with doing", from Ursula K. LeGuin's Earthsea novels, always struck me as such a powerful phrase. An entire state of mind boiled down to 5 words. But then again I remember her saying eastern philosophy greatly influenced her writing, if I'm not mistaken

Re: Zen and the Art of Machine Learning Research

#17

Post by johndhi »

peepee1982 wrote: post #9
jdw64 wrote: post #4

I feel that the Zen used in the West and the Zen in East Asia are quite different. I think the Western Zen is probably the one from the 1970s book Zen and the Art of Motorcycle Maintenance. It usually carries a sense of equanimity and beginner's mind. But in East Asia, Zen actually emphasizes aimlessness or non‑purposefulness. The point where I really feel the difference is that Western Zen seems to be about how to t…

Similarly, the Western idea of Stoicism seems to focus mostly on controlling or even suppressing your emotions (at least on surface level), while the Stoicism you rightly call "Roman" (thanks for that, btw) is much more holistic and more of an ethical framework.

Who doesn't call stoicism Roman?

Re: Zen and the Art of Machine Learning Research

#18

Post by sph »

jdw64 wrote: post #4

I feel that the Zen used in the West and the Zen in East Asia are quite different. I think the Western Zen is probably the one from the 1970s book Zen and the Art of Motorcycle Maintenance. It usually carries a sense of equanimity and beginner's mind. But in East Asia, Zen actually emphasizes aimlessness or non‑purposefulness. The point where I really feel the difference is that Western Zen seems to be about how to t…

> Zen actually emphasizes aimlessness or non‑purposefulness

The visual metaphor from Taoists is being like 'uncarved wood'. Western Zen has been bastardised and commercialised, whereas one can look into Taoism to find many of the same concepts that, by virtue of their own simplicity, have remained timeless. The "problem", so to speak, with Zen is being associated with Buddhism, which has a long and intricate history and body of works attached to it, yet moves towards the same line of simplicity and spontaneity of Taoism.

In the words of Alan Watts, it all starts with the eternal Tao; all other religions are for people that need the same ideas overcomplicated with too many words.

Re: Zen and the Art of Machine Learning Research

#20

Post by keybored »

jdw64 wrote: post #4

I feel that the Zen used in the West and the Zen in East Asia are quite different. I think the Western Zen is probably the one from the 1970s book Zen and the Art of Motorcycle Maintenance. It usually carries a sense of equanimity and beginner's mind. But in East Asia, Zen actually emphasizes aimlessness or non‑purposefulness. The point where I really feel the difference is that Western Zen seems to be about how to t…

Given AI’s impact on society, I read this more as Zen And The Practice of Kamikaze.
Post reply on HN