Ugh I’m beginning to think I’m going to spend the next 6-12 months commenting “no, large language models aren’t supposed to somehow know everything in the world. No, that’s not what they’re designed for. Yes, hooking one up to our long-standing record-of-everything-in-the-world (google’s knowledge graph) is going to be powerful.” It’s getting to point where I need to consider stop going on HN. This is like when my fa…
I understand why the mainstream thinks that but it's incredibly annoying that even in tech circles there is very little meaningful discussion, it's mainly just people posting amusing screenshots purportedly showing how smart GPT3 is or in other cases how it's politically biased. Anyone who's played around with it knows that it's fun but it's not a search engine replacement and it doesn't know nor understand things. I…
For myself, I tried GPT models and read the Attention Is All You Need paper before ChatGPT. I also read analysis, for example, from Gwern, about capability overhangs and underestimation of present capabilities in these models. In many cases, I found myself agreeing with the logic. I found that it was very possible to coax greater capabilities out of the model than many presumed them to have. I still find this to be the case and have, in recent memory demonstrated this is true in present models: for example, I posted a method of coaxing the solving of some puzzles by prompting to include the representation of intermediate states in order to successfully solve problems related to reasoning puzzles of a 'can contain' nature, which was a capability that someone claimed these language models lack, despite them gaining that capability when appropriately prompted, which suggests that they always had that capability in their weights, but that it wasn't exercised successfully - the capability was there, but not used, rather than absent, as claimed by the people who claimed it was absent.
That said, I don't think it matters much what most people did or didn't do with regard to this experimentation and, as you imply, ages really did past - I would feel trepidation, not hope, about the quality of my ideas compared to the people who came later. Historically, the passing of ages tends to improve, not diminish. So if I was experimenting with, for example, flying machines in the 1700s, but then ages past and someone who did not do that experimenting was talking to me about flying machines in the early 2000s, I would suspect them to be more informed, not less informed, than I was. They, as a matter of course in casual classroom settings, have probably done better than my best experiments including high effort costly experiments. Their toys fly. A generation ago, we would talk about planes, but now we can also talk about their toys. It is that normal to them. They have so much better priors.