New technology could blow away GPT-4 and everything like it
1–10 of 18 posts
Re: New technology could blow away GPT-4 and everything like it
#2I had so much fun with CNN models just before BERT hit it big. It would be nice to see them make a comeback.
Re: New technology could blow away GPT-4 and everything like it
#3I didn't see anything in the article about what the scaling factor was? less than P^2 but what was it?
Re: New technology could blow away GPT-4 and everything like it
#4I didn't see anything in the article about what the scaling factor was? less than P^2 but what was it?
The paper has a “preliminary scaling law” diagram. The shape of the graph is the same, but with 20% fewer FLOPS.
The real breakthrough is that Hyena apparently has an unlimited context window.
Re: New technology could blow away GPT-4 and everything like it
#5> At 64,000 tokens, the authors relate, "Hyena speed-ups reach 100x" -- a one-hundred-fold performance improvement.
That’s quite the difference
Re: New technology could blow away GPT-4 and everything like it
#6I didn't see anything in the article about what the scaling factor was? less than P^2 but what was it?
presumably still O(n2) in theory, but not for practical cases.
I think that anything reolacing attention will suffer quadratic growth for some pathological examples.
maybe if we have a better understanding of the data we could give a better definition (much like graph complexity is usually given in the actual number of edges, which are theoretically O(n2).)
Re: New technology could blow away GPT-4 and everything like it
#7Blog post: https://hazyresearch.stanford.edu/blog/2023-03-07-hyena
Previous discussion: https://news.ycombinator.com/item?id=35502187
Re: New technology could blow away GPT-4 and everything like it
#8I didn't see anything in the article about what the scaling factor was? less than P^2 but what was it?
The paper has a “preliminary scaling law” diagram. The shape of the graph is the same, but with 20% fewer FLOPS. The real breakthrough is that Hyena apparently has an unlimited context window.
>The real breakthrough is that Hyena apparently has an unlimited context window.
It's extrapolated volition time (ﾟ∀ﾟ)
Re: New technology could blow away GPT-4 and everything like it
#9> At 64,000 tokens, the authors relate, "Hyena speed-ups reach 100x" -- a one-hundred-fold performance improvement. That’s quite the difference
Classic attention is quadratic in context length and faster alternatives seem to not perform as well, wonder how Hyena is in comparison to linear attention algorithms.
Re: New technology could blow away GPT-4 and everything like it
#10How good is it at scaling? And will it still retain the emergent capabilities of the huge transformer LLMs?
Isn't this basically the bitter lesson again? Making small improvements work but in long term it won't give the same impressive result?