101–110 of 227 posts
Graduate Student Descent -- there's also the other saying: Every moment you spend writing, publishing, or working on your dissertation is a moment that you are getting behind on doing unique and cutting edge research.
But also "The difference between screwing around and science is writing it down."
There's a nice talk by Yann LeCun where he goes on to explain really well why deep learning has such fast progress. [0] He goes on to explain how theory always comes later. I thought that information theory came before practice but turns out it also came after. (there was a bunch of heuristics for sending messages with teletypes) A nice example is Roman technological advances in architecture and materials that predat…
Earlier quoted context omitted.
There somehow seems to be an unwritten law of the Python generation like "Thou shalt not criticise Machine Learning". Or is there a better explanation for the emotions that flare up every time someone dampens the exaggerated expectations and reminds us of earlier research in the field of linguistics or AI?
Not your problem. Just let 'em be stuck in their local optimum (probably better this way) :P
Unfortunately I'm not unaffected.
Earlier quoted context omitted.
There somehow seems to be an unwritten law of the Python generation like "Thou shalt not criticise Machine Learning". Or is there a better explanation for the emotions that flare up every time someone dampens the exaggerated expectations and reminds us of earlier research in the field of linguistics or AI?
There's plenty of good criticism of given products, papers, and approaches. There's also bad criticism though and it's important to distinguish the two.
I fear ML and AI are taking the lion share of analytics resources in most big companies and that's just not right. We still have basic problems with data integrity and blending. Getting a real end to end view of any business process is still a herculean task. Try matching something like invoices with marketing leads, good luck...
If you want a reporting solution, buy AI.
Some good points in the article, but I disagree with the tone and the conclusion. > we’ve rewarded and lauded incremental researchers as innovators, increased their budgets so they can do even more incremental research There isn't a scientific field where every single paper is groundbreaking. It's a Brownian motion of small incremental innovations, until eventually we stumble upon something big (like deep learning).…
Some good points in the article, but I disagree with the tone and the conclusion. > we’ve rewarded and lauded incremental researchers as innovators, increased their budgets so they can do even more incremental research There isn't a scientific field where every single paper is groundbreaking. It's a Brownian motion of small incremental innovations, until eventually we stumble upon something big (like deep learning).…
Is the deep learning really the result of incremental research? The SOTA chasing frenzy comes after the discovery of deep learning. The motivation of incremental research can hardly be justified as to discover the next deep learning, although they might do.
Some good points in the article, but I disagree with the tone and the conclusion. > we’ve rewarded and lauded incremental researchers as innovators, increased their budgets so they can do even more incremental research There isn't a scientific field where every single paper is groundbreaking. It's a Brownian motion of small incremental innovations, until eventually we stumble upon something big (like deep learning).…
> A nitpick: CNNs are most definitely not a generalization of matrix multiplication. In fact, the opposite: you can view CNNs as a matrix multiplication with a particular matrix structure.
Neither is really a generalization. Any matrix multiplication can be implemented with a convolution function, and yet any convolution can be represented as a matrix multiplication via im2col.
This is what I finally found: BERT, which stands for Bidirectional Encoder Representations from Transformers.
And, while I agree there is a "fake rigor" problem in ML research, the particular examples that they bring up aren't extremely good exemplars in my opinion. Instead, they seem to have a problem with the standard operating procedure of mathematics while missing the point that it is what got the "stack more layers" school here in the first place. Simplifying a problem so you can understand it, and then relaxing the assumptions and seeing if you can figure out what implications that has is how advances are made.
Plus, they have some hot takes and statements that are just plain wrong.
> With Automatic Differentiation, the backward pass is essentially free and is as engaging to compute as 50 digit number long division. Deriving long complicated gradients is fake rigor that was useful before we had computers.
What? First of all, AD is not a solved problem and using it is not "essentially free." There's a huge performance overhead when adding AD to a system. Try using a second order method with AD. I hope your Hessian actually finished computing.
> Julia on the other hand is a language made for scientific computing where everything is automatically differentiable by default.
This is just plain false. I'm a huge proponent of Julia, and there are some great AD packages, but in no way is everything automatically differentiable (even with the nice packages), nor is that a design goal. The work on Flux.jl (a package) is extremely impressive though, and there are particular features of Julia that allow some awesome package interoperability (e.g. the fact that some ODE solvers can be differentiated through with Flux).