31–40 of 96 posts
The MAMBA [1] model gained some traction as a potential successor. It's basically an RNN without the non linearity applied across hidden states, which makes it logarithmic time (instead of linear time) inference with a parallelizable scan [2]. It promises much faster inference with much lower compute costs, and I think up to 7B params, performs on par with transformers. I've yet to see a 40B+ model trained. The resea…
I hate that popular domains take ownership of highly generic words. Many years ago, I struggled for a while to understand that when people say "frontend" they often mean a website frontend, even without any further context.
The worst offender is "feature". In my domain (ML and geo) we have three definitions. Feature could be referring to some addition to the user-facing product, a raster input to machine learning, or a vector entity in GeoJSON. Context is the only tool we have to make the distinction, it gets really confusing when you're working on features that involve querying the features with features.
1) an architecture described in a paper
2) the trained weights of a specific instantiation of architecture
3) a chunk of code/neural net that accomplishes a task, agnostic to the above definitions
Right now as long as the rocket's heading straight up, everyone's on board with MLPs (Multilayer Perceptrons/Transformers)! Why not stay on the same rocket for now!? We're almost at AGI already!
Yes. Happy to chat if u msg me. Using RL coupled with NNs to integrate search directly into inference instead of as an afterthought like Chain of though and test time training.
Earlier quoted context omitted.
https://en.m.wikipedia.org/wiki/Transformer_(deep_learning_a...
Yeah, it's literally the most important practical development in AI/ML of the decade. This is like reading an article (or headline, more like) on HN and saying "please define git".
It's really not. "Git" has a single extremely strong definition for tech people, and a single regional slang definition. "Transformer" has multiple strong definitions for tech people, and multiple strong definitions colloquially.
Not that we can't infer the OP's meaning - just that it's nowhere near as unambiguous as "git".
You have stuff like: https://www.literal-labs.ai/tsetlin-machines/ and https://tsetlinmachine.org/ European initiatives..
52x less energy is crazy. Seems like it's in the veeery early stages, though, a quick search basically only yields the original paper and articles about it. This comment from the creator really shines light on the novel approach, though, which I find oddly antagonistic towards Big Tech: "Where the Tsetlin machine currently excels is energy-constrained edge machine learning, where you can get up to 10000x less energy…