Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Exponentially faster language modelling

arxiv.org

Post reply on HN
Print view View original

81–90 of 150 posts

Re: Exponentially faster language modelling

#81

Post by godelski »

Micoloth wrote: post #29
WithinReason wrote: post #24

Earlier quoted context omitted.

Also, this didn't come from OpenAI or DeepMind, or even industry. What are those guys even doing? :)

They sure as hell have no incentives to make Neural Network faster and more accessible, for starters.. (Considering they right now make more money and have more control, the less accessible and the more computation-hungry AI models are) To be fair, this approach (claims to) only speed up inference, not training, so all the GPUs are needed anyway.

I wouldn't be so quick to conspiracy. I'm the author of a work and a famous blog post that trains a particular common architecture much faster (don't want to dox myself too much) and with far fewer parameters, but it has been rejected several times and is now arxiv only. Our most common complaint was "who would use this? Why not just take a large model and tune it?" That question alone held us back a year (had over a hundred citations by then and remains my most cited work) until it switched to "use more datasets" and "not novel" (by that time true, others had built off of us, cited us, and published in top venues).

I don't think this was some conspiracy by big labs to push back against us (we're nobodies) but rather that people get caught up in hype and reviewers are lazy and incentivized to reject. You're trained to be critical of works and especially consider that post hoc most solutions appear far simpler than they actually are. But context matters because if you don't approach every paper with nuance it's easy to say "oh, it's just x." But if those ideas were so simple and obvious they would also be prolific. I see a lot of small labs suffer the same fate simply due to lack of compute. If you don't make your new technique work on many datasets it becomes the easiest thing to reject a paper by. ACs aren't checking that reviews are reasonable. I've even argued with fellow reviewers about papers in workshops -- papers I would have accepted in the main conference -- that are brushed off and the reviewers admit in their reviews that they do not work on these topics. I don't understand what's going on but at times it feels like a collective madness. A 10 page paper with 4 very different datasets that solves a problem, is clearly written, has no major flaws, and is useful to the community should not need defending when submitted to a workshop just because reviewers aren't qualified to review the work (this paper got in btw). We are moving into a "pay to play" ecosystem and that will only create bad science due to group think. (another aspect of "pay to play" is in the tuning. Spending $1M to tune your model to be the best doesn't mean it is better than a model that could not afford the search. Often more than half of resources are spent on tuning now)

Re: Exponentially faster language modelling

#82

Post by rsolva »

I find running 7B models on my 6 year old small form factor HP EliteDesk to be fast enough for casual everyday use. If this speedup can be applied generally to commonly used models, I can serve a local ChatGPT experience for both friends and family from my tiny homelab in my basement.

mind blown

Re: Exponentially faster language modelling

#83

Post by sroussey »

entropicdrifter wrote: post #69
marcinzm wrote: post #18

Earlier quoted context omitted.

Bottleneck for larger models however this would presumably allow for cheaper models at scale or on compute constrained devices (like phones).

And potentially for distributing a model across several devices at inference time. You could devote a cluster of smaller/weaker machines to inference.

You can do that today, the only advantage today though is being able to fix the model in memory. It’s sequential and slower due to communication costs, though batching might be faster?

Re: Exponentially faster language modelling

#84

Post by godelski »

lawlessone wrote: post #45
WithinReason wrote: post #30

Earlier quoted context omitted.

They certainly have an incentive to keep these kinds of improvements in-house and not publish them, since they are commercial entities and this represents a competitive advantage.

I think Nvidia might have an incentive for this not to exist. edit: but you are right for the AI companies not open sourcing their models it's an advantage to have it when others don't

Nvidia can't make GPUs fast enough. I doubt 10xing training and/or inference efficiency would result in a decrease in demand. I would be surprised if it didn't instead increase demand. Mind you, Nvidia is pushing hard on TensorRT which optimizes models at inference time and results in major increases in throughput (not 10x though lol).

Re: Exponentially faster language modelling

#85

Post by lopuhin »

joelthelion wrote: post #70
lopuhin wrote: post #66

Earlier quoted context omitted.

There are two issues here -- for one, in big transformers, more compute is in the attention layers, while this work improves only feed-forward layers, which are more important for smaller models and smaller sequence lengths. Second, in many typical scenarios LLM inference is memory bandwidth bound, I'm not sure if it's possible to utilize their approach to reduce required memory bandwidth.

Doesn't reducing the number of neurons drastically reduce memory requirements?

Yes it might. "Reduction of number of neurons" is not static here, unlike traditional pruning approaches, here they still keep all weights, but the network dynamically selects which sub-portion of them to use. There is a related discussion of this in section 3.2 (page 4), but they don't think they mention actual memory bandwidth requirements/wins of their implementation, and probably there can be different tradeoffs for different devices.

Re: Exponentially faster language modelling

#87

Post by valine »

yvdriess wrote: post #11
Klaster_1 wrote: post #6

What are the potential consequences? Does this open doors to faster edge inference or improved capabilities?

Both. Cheaper CPU-based inference, GPUs are not as competitive for sparse linear algebra. This could lead to much larger models, as you only touch a small portion of the matrix during inference. However, the training here is still dense-LA on a GPU, so you still blow up the compute cost when increasing model size.

GPU utilization should be down when using this technique. I’m hoping this could allow for more efficient batch inference on GPUs. If you can predict 10 tokens for the price of 1 it should allow you to do tree of thought much more efficiently.

https://github.com/princeton-nlp/tree-of-thought-llm

Re: Exponentially faster language modelling

#89

Post by foobiekr »

bloopernova wrote: post #80
WithinReason wrote: post #48

Earlier quoted context omitted.

That's an interesting question. It actually provides a nice way to parallelized training: Pretrain e.g. the first 3 branch levels, which effectively fragments the model into 8 separate parts, which you can continue training across 8 independent servers/nodes with no further communication between the nodes. A central server would run the 1st 3 levels and mark parts of the training set that each node has to train on. M…

Apologies for layman question: how much tera/peta/exa-flops do current models use to train? Well, I'm assuming they'd use whatever they're given, so maybe the question should be "how much less time would training take on a 1.5 exaflops computer?"

As many as they can afford.

A lot of clusters are totally homogeneous, at least within some very large domains, so for a given interconnect and a generation of GPU you know the maximum message latency, the peak sustained pflop rate, and so on but what often matters is some combination of the depreciation-cost-per-time and the watt hours per unit time, where you can sort of approximate both if you ignore the unfortunate realities, which then act as a multiplier.

For example, a problem is network issues - and not just scale - as the training sequence often involve billions of cycles of short compute-sync sequences which are bursty (e.g., all-to-all, barrier, compute, barrier, all to all, ...) but between which there isn't enough time to engage low power modes so you're burning $ due to slack and waste. This is true in different ways for a lot of training approaches.

You can approximate this, but it's so sensitive to data set size, specific training schedule, etc. that you won't be able to get the most important answer.

Re: Exponentially faster language modelling

#90

Post by EForEndeavour »

infecto wrote: post #88
jafitc wrote: post #86

[flagged]

Not a fan of what looks to be an AI summary in the comments.

Yep. And we only notice the obvious ones. And the quality of AI comments will only improve over time. Which raises the question (unanswered as far as I can tell) of what to do about it, and whether it even really matters...
Post reply on HN