11–20 of 37 posts
Reminds me of the classic paper "The Effects of Moore's Law and Slacking on Large Computations": https://arxiv.org/abs/astro-ph/9912202
Also makes you think.
Earlier quoted context omitted.
I’m sorry … but what?! As someone who worked on early LLM tech at Google hundreds of millions were poured into it. Do you think that paper just spontaneously came from pure theory?
Most of the core idea of transformers was invented in the early 1990s, including what at the time were termed Fast Weight Programmers and are formally equivalent to linearised self-attention: https://people.idsia.ch/~juergen/fast-weight-programmer-1991... . Google just had the hardware to actually run and experiment with large transformers.
That is the investment and it was no small price. JUST is carrying a ton of weight here.
This completely misses the concept that the technology doesn’t just create itself without investment. SOMEONE has to invest for that progress to be made. It’s why I always hated the application of this to tech.
> SOMEONE has to invest for that progress to be made. History often has cases where the surrounding context/infrastructure/market/culture just wasn't suitable for an invention to take off--so it fails--and then somebody 5-100 years later tries something almost the same thing and it becomes a wild success and they get all the credit for it. For example, look at the AT&T Picturephone from ~1964. [0] Even if someone fro…
This completely misses the concept that the technology doesn’t just create itself without investment. SOMEONE has to invest for that progress to be made. It’s why I always hated the application of this to tech.
The wait equation is totally irrelevant here. Where the wait equation has relevance is the fact that a generation ship, once launched, is likely unable to reconfigure itself to take advantage of new technologies invented while it is underway. This is key to why there is an incentive to wait, and there is no parallel in the idea of investing in AI. Someone who chooses to invest in AI does get access to tech that devel…
Well duh, but first find all those coefficients and then come back and we'll talk yeah?
I have so many problems with this line
Earlier quoted context omitted.
> SOMEONE has to invest for that progress to be made. History often has cases where the surrounding context/infrastructure/market/culture just wasn't suitable for an invention to take off--so it fails--and then somebody 5-100 years later tries something almost the same thing and it becomes a wild success and they get all the credit for it. For example, look at the AT&T Picturephone from ~1964. [0] Even if someone fro…
But we did people continuously invested through the 80s then 90s then Cisco then Google and now zoom. Do you and the writer not know that video calling was a consistent investment for over 40+ years?!
Just because something else in civilization might somehow help someday isn't enough to make it part of "the investment" which a company or individual is deciding whether to make or abandon for a different goal they are focused on.