111–120 of 138 posts
Earlier quoted context omitted.
To know, you must perform intellectual work, not merely be smart. I bet you are smart enough.
What a nice comment!! This has been a big failing of my mental model. I always believed if I was smart enough I should understand things without effort. Still trying to unlearn this....
Earlier quoted context omitted.
Same here. I blame the popularity of Next.js. More and more of the web is slowly becoming more broken on Firefox on Linux, all with the same tired error: "Application error: a client-side exception has occurred"
Works fine on Firefox on Linux for me.
Next.js was never really broken for Firefox for Linux in my experience.
Through some "hidden" settings, disabling JS, and proably quite many plugins can brake things.
The only thing which tends to be often "broken" for FF in my experience is often CORS and Mic/Camera APIs, ironically in case of CORS 100% because of bugs in non standard compliant websites and for Mic/Camera it's more complicated but most common websites simply refusing to work with FF without even trying (and if you trick them into believing it's no FF often working just fine...).
Also, speaking about transformers: they usually append their output tokens to input and process them again. Can we optimize it, so that we don't need to do the same calculations with same input tokens?
Earlier quoted context omitted.
I find this problem fascinating. For decades we’ve puzzled at how the inner workings of the brain works, and thought we’ve learned a lot we still don’t fully understand it. So, we figure, we’ll just make an artificial brain and THEN we’ll be able to figure it out. And here we are, finally a big step closer to an artificial brain and once again, we don’t know how it works :) (Although to be fair we’re spending all of…
Is it a brain, though? As far as I understand it it's mostly stochastic calculations based on language or image patterns whose rule sets are static and immutable. Every conceivable idea of plasticity (and with that: a form of fake consciousness) is only present during training. Add to that the fact that a model is being trained actively and the weights are given by humans and the possible realm of outputs is being he…
Earlier quoted context omitted.
To know, you must perform intellectual work, not merely be smart. I bet you are smart enough.
What a nice comment!! This has been a big failing of my mental model. I always believed if I was smart enough I should understand things without effort. Still trying to unlearn this....
Understanding transformers will be really hard if you don't understand basic fully connected feedforward networks (multilayer perceptrons). And learning those is a bit challenging if you don't understand a single unit perceptron.
Transformers have the additional challenge of having a bit weird terminology. Keys, queries and values kinda make sense from a traditional information retrieval literature but they're more a metaphor in the attention system. "Attention" and other mentalistic/antrophomorphic terminology can also easily mislead intuitions.
Getting a good "learning path" is usually a teacher's main task, but you can learn to figure those by yourself by trying to find some part of the thing you can get a grasp of.
Most complicated seeming things (especially in tech) aren't really that complicated "to get". You just have to know a lot of stuff that the thing builds on.
Damn, this looks phenomenal. I've been wanting to do a deep dive like this for a while-- the 3D model is a spectacular pedagogic device.
Andrej Karpathy twisting his hands as he explains it is also a great device. Not being sarcastic, when he explains it I understand it for a good minute it two. Then need to rewatch as I forget (but that is just me)!
This is a great visualization because original paper on transformers is not very clear and understandable; I tried to read it first and didn't understand so I had to look for other explanations (for example it was unclear for me how multiple tokens are handled). Also, speaking about transformers: they usually append their output tokens to input and process them again. Can we optimize it, so that we don't need to do t…
Earlier quoted context omitted.
I find this problem fascinating. For decades we’ve puzzled at how the inner workings of the brain works, and thought we’ve learned a lot we still don’t fully understand it. So, we figure, we’ll just make an artificial brain and THEN we’ll be able to figure it out. And here we are, finally a big step closer to an artificial brain and once again, we don’t know how it works :) (Although to be fair we’re spending all of…
Is it a brain, though? As far as I understand it it's mostly stochastic calculations based on language or image patterns whose rule sets are static and immutable. Every conceivable idea of plasticity (and with that: a form of fake consciousness) is only present during training. Add to that the fact that a model is being trained actively and the weights are given by humans and the possible realm of outputs is being he…