71–80 of 108 posts
Earlier quoted context omitted.
How does the EU feature in TFA exactly?
There seems to be a coordinated push around Schmidhuber all around media in the EU, even LinkedIn is full of "random" posts about him in the past week.
There's this crowd on HN which is very vocal against academia. From what I've seen, the main points are that academia isn't efficient, most of the science coming out of academia is useless and that the whole system is just a waste of taxpayers money. Instead, what is often argued, all good research is done in private labs. Then pointing to SpaceX, Moderna, OpenAI, Google, etc. And while it is very true that often the…
Like half of what Schmidhuber is always complaining about is that (except for LSTMs) people aren't standing on the shoulders of his research very much. They try to solve some of the same problems people have always wanted to solve, try some of the same approaches people always tend to try, and then tinker until it works. At no point do they consult Schmidhuber's decade-old papers where he tried something kind of simi…
To put it more simply, people with academic credentials should not demand acknowledgement of their current intellectual work while denigrating and ridiculing the importance of very similar work done in the past.
Also see Schmidhuber's take on the Hinton + Hopfield Nobel prize: https://people.idsia.ch/~juergen/physics-nobel-2024-plagiari...
It's sad that he is the only one speaking out about Hinton. This whole Hinton glorification seems like it's being pushed by an agenda. I'm not sure if he would receive this much attention if he held a different view (closer to LeCun or Ng), rather than these Effective Altruism takes on current AI.
It's crazy to think that if Elon Musk hadn't mentioned Schmidhuber, most people would have no idea. It's nauseating how all the researchers who happened to work for big tech got tons of media coverage but Schmidhuber and his team were getting zero coverage yet they made massive contributions. I bet there are many others not mentioned. Nobody even knows about Frank Rosenblatt. It's insane how distorted our perception…
> Nobody even knows about Frank Rosenblatt. Very Trump-like statement - "Not many people know this, but ...". Yes, I lot of people know this. Any class that even says a little about the history of NNs will talk about Rosenblatt and the Perceptron.
Sure. I think it starts to get more interesting when the influences that Rosenblatt explicitly cites in his seminal Perceptron paper (e.g. Hayek) become part of the discussion (which rarely happens in my experience).
worth separating: LSTM (Hochreiter & Schmidhuber 1997) is ironclad and widely cited. the transformer attention priority claims are far shakier. conflating them is how Schmidhuber undermines himself
https://www.youtube.com/watch?v=AIiwuClvH6k
When it comes to attention, details matter, since the idea itself is obvious - weighted inputs, and implicit attention is present in every neural network - this is what weights are.
The specific form of attention used by the Transformer is key-based associative attention, aka "Bahdanau attention" introduced in Bahdanau's paper "Neural Machine Translation by Jointly Learning to Align and Translate". It's perhaps worth noting that the word "attention" is barely even mentioned in this paper, other than noting that this weighted input mechanism can be seen as a form of attention (presumably mentioned since attention was at that time a recurring theme in various types of neural network).
Bahdanau attention - not just the general concept of attention - seems to be a very critical piece of the Transformer architecture since this this is what allows the Transformer to find things in context and is behind the "induction head" mechanism that appears central to how Transformers operate.
Earlier quoted context omitted.
I jest but database design is its own sub field of computer science, maybe look into their papers?
I did that too. Ending up building my own reverse index with a fixed-size vocabulary. But that's my issue, you start building one product and you end-up building ten in the process to solve all edge cases because no one bothered to research how things scale.
There's this crowd on HN which is very vocal against academia. From what I've seen, the main points are that academia isn't efficient, most of the science coming out of academia is useless and that the whole system is just a waste of taxpayers money. Instead, what is often argued, all good research is done in private labs. Then pointing to SpaceX, Moderna, OpenAI, Google, etc. And while it is very true that often the…
I do a lot of work that is based on academic research, aka building a proprietary sparse embedding model. My issue with academia is that they don’t bother to solve the practical issues. They tell you how to build a PPMI model, but what about hitting a database that’s 500TB to find co-occurrence numbers? This isn’t even touched so you’d then have to go and invent a bazillion of algorithms yourself to make your life ea…
Earlier quoted context omitted.
[flagged]
well, so you think, all parts and peoples of USSR were voluntarily part of USSR?
And do you think the people in Europe are voluntarily a part of your undeclared, but very much existing, empire?
Earlier quoted context omitted.
> From what I've seen, the main points are that academia isn't efficient, most of the science coming out of academia is useless and that the whole system is just a waste of taxpayers money. Instead, what is often argued, all good research is done in private labs. Then pointing to SpaceX, Moderna, OpenAI, Google, etc. Well... that's "starve the beast" in action. A lot of things we take for granted, that underpin our m…
Reusable rockets were not invented by nasa because their mission was exploration not commercialization.
https://en.wikipedia.org/wiki/Jet_Propulsion_Laboratory
> Founded in 1936 by California Institute of Technology (Caltech) researchers, the laboratory is now owned and sponsored by NASA and administered and managed by Caltech.
Minimum-Landing-Error Powered-Descent Guidance for Mars Landing Using Convex Optimization http://larsjamesblackmore.com/BlackmoreEtAlJGCD10.pdf
Elon originally wanted parachutes and was convinced by Lars to go with self landing rockets.