Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Exponentially faster language modelling

arxiv.org

Post reply on HN
Print view View original

91–100 of 150 posts

Re: Exponentially faster language modelling

#91

Post by dragonwriter »

EForEndeavour wrote: post #90
infecto wrote: post #88

Earlier quoted context omitted.

Not a fan of what looks to be an AI summary in the comments.

Yep. And we only notice the obvious ones. And the quality of AI comments will only improve over time. Which raises the question (unanswered as far as I can tell) of what to do about it, and whether it even really matters...

IMO:

If the comment is a quality comment, I don't care if the poster used AI, if its a low-quality or inappropriate comment (and a summary of the existing comments counts), then downvote or, for egregious cases, flag it.

Re: Exponentially faster language modelling

#92

Post by thomasahle »

WithinReason wrote: post #14

Link to previous paper: https://arxiv.org/abs/2308.14711 An attempt at a summary: They use a sigmoid function to make differentiable "soft" branches, and stack them to construct a binary tree, with the goal of only taking one branch at inference time (but training the whole tree) leading to log(W) instead of W inference cost. They gradually harden the branches so they become hard branches at the end of training. A br…

Sounds like hiarchial softmax from the early NLP days

Re: Exponentially faster language modelling

#93

Post by klyrs »

richardw wrote: post #72
WithinReason wrote: post #48

Earlier quoted context omitted.

That's an interesting question. It actually provides a nice way to parallelized training: Pretrain e.g. the first 3 branch levels, which effectively fragments the model into 8 separate parts, which you can continue training across 8 independent servers/nodes with no further communication between the nodes. A central server would run the 1st 3 levels and mark parts of the training set that each node has to train on. M…

Also steps up the economic benefit of, and therefore demand for, botnets. We really need a solution to bad actors controlling vast amounts of compute.

That ship has sailed, and her name is bitcoin.

Re: Exponentially faster language modelling

#94

Post by thomasahle »

dragonwriter wrote: post #91
EForEndeavour wrote: post #90

Earlier quoted context omitted.

Yep. And we only notice the obvious ones. And the quality of AI comments will only improve over time. Which raises the question (unanswered as far as I can tell) of what to do about it, and whether it even really matters...

IMO: If the comment is a quality comment, I don't care if the poster used AI, if its a low-quality or inappropriate comment (and a summary of the existing comments counts), then downvote or, for egregious cases, flag it.

The issue (temporarily) is that many consider a long comment = high effort = probably high quality. Now we have to change our heuristics.

Re: Exponentially faster language modelling

#95

Post by richardw »

klyrs wrote: post #93
richardw wrote: post #72

Earlier quoted context omitted.

Also steps up the economic benefit of, and therefore demand for, botnets. We really need a solution to bad actors controlling vast amounts of compute.

That ship has sailed, and her name is bitcoin.

If bitcoin keeps the botnets away from world beating AI, worth it.

Re: Exponentially faster language modelling

#97

Post by FooBarWidget »

rsolva wrote: post #82

I find running 7B models on my 6 year old small form factor HP EliteDesk to be fast enough for casual everyday use. If this speedup can be applied generally to commonly used models, I can serve a local ChatGPT experience for both friends and family from my tiny homelab in my basement. mind blown

I find 7B models to be too stupid. They often respond with nonsense or fail to follow instructions.

Re: Exponentially faster language modelling

#99

Post by kristianp »

iNic wrote: post #10
ndr wrote: post #4

Abstract: > Language models only really need to use an exponential fraction of their neurons for individual inferences. As proof, we present UltraFastBERT, a BERT variant that uses 0.3% of its neurons during inference while performing on par with similar BERT models. UltraFastBERT selectively engages just 12 out of 4095 neurons for each layer inference. This is achieved by replacing feedforward networks with fast fee…

Do I understand correctly that the difficulty of making this useful is writing code to run this idea on GPUs?

> a PyTorch implementation delivering 40x speedup over the equivalent batched feedforward inference

Does this not indicate a 40x speedup on the GPU?

Edit: looking at the paper, their "Naive CUDA" implementation also shows a 117x speedup in Table 2.

Post reply on HN