Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Exponentially faster language modelling

arxiv.org

Post reply on HN
Print view View original

111–120 of 150 posts

Re: Exponentially faster language modelling

#111

Post by sorokod »

GaggiX wrote: post #110
FooBarWidget wrote: post #97

Earlier quoted context omitted.

I find 7B models to be too stupid. They often respond with nonsense or fail to follow instructions.

Even like OpenChat-3.5? (Probably the best 7B model out there) Demo: https://openchat.team/ HuggingFace: https://huggingface.co/openchat/openchat_3.5 On the LLM arena (blinded comparisons), it's the third best non-proprietary model: https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboar...

What is the sum of odd numbers in this set: 4, 7, 12, 1, 3

The sum of odd numbers in the given set is 4 + 7 + 1 = 12. Therefore, the answer is 12.

Re: Exponentially faster language modelling

#112

Post by quickthrower2 »

WithinReason wrote: post #24
WithinReason wrote: post #14

Link to previous paper: https://arxiv.org/abs/2308.14711 An attempt at a summary: They use a sigmoid function to make differentiable "soft" branches, and stack them to construct a binary tree, with the goal of only taking one branch at inference time (but training the whole tree) leading to log(W) instead of W inference cost. They gradually harden the branches so they become hard branches at the end of training. A br…

Also, this didn't come from OpenAI or DeepMind, or even industry. What are those guys even doing? :)

It is not surprising. The assumption is that they have the best people. That you can objectively search 8 billion people for the best people globally is folly of course. There are geniuses without US citizenship / visas / green cards. And so outside brains are going to figure this out. Mix in GDP of $rest_of_world has much more resources than any company, and the luck-driven nature of making AI discoveries, and I reckon most progress will be outside of OpenAI etc. Driven by a problem the big guys don't need to solve: how do I avoiding buying a $5k graphics card.

Re: Exponentially faster language modelling

#113

Post by rictic »

godelski wrote: post #84
lawlessone wrote: post #45

Earlier quoted context omitted.

I think Nvidia might have an incentive for this not to exist. edit: but you are right for the AI companies not open sourcing their models it's an advantage to have it when others don't

Nvidia can't make GPUs fast enough. I doubt 10xing training and/or inference efficiency would result in a decrease in demand. I would be surprised if it didn't instead increase demand. Mind you, Nvidia is pushing hard on TensorRT which optimizes models at inference time and results in major increases in throughput (not 10x though lol).

Yeah, Jevons Paradox suggests that 10xing efficiency of training and inference would increase demand for GPUs.

Re: Exponentially faster language modelling

#114

Post by all2 »

sorokod wrote: post #111
GaggiX wrote: post #110

Earlier quoted context omitted.

Even like OpenChat-3.5? (Probably the best 7B model out there) Demo: https://openchat.team/ HuggingFace: https://huggingface.co/openchat/openchat_3.5 On the LLM arena (blinded comparisons), it's the third best non-proprietary model: https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboar...

What is the sum of odd numbers in this set: 4, 7, 12, 1, 3 The sum of odd numbers in the given set is 4 + 7 + 1 = 12. Therefore, the answer is 12.

Technically 3 is even. \s

Re: Exponentially faster language modelling

#115

Post by anonymousDan »

singhrac wrote: post #57
swalsh wrote: post #33

Earlier quoted context omitted.

Has anyone used SIMD instructions to try and speed up cpu inference?

A lot of CPU inference libraries (llama.cpp included) use as much SIMD as possible, sometimes by hand-writing loops. The one I hack on, llama.rs, uses portable_simd but specializes to your CPU at compile time. My experience has been that most CPU inference is actually not compute limited, but memory bandwidth limited, since most weights are used for a few operations per token (how quickly can you load and unload the…

Would you say that is the state of the art CPU inference library?

Re: Exponentially faster language modelling

#116

Post by moffkalast »

GaggiX wrote: post #110
FooBarWidget wrote: post #97

Earlier quoted context omitted.

I find 7B models to be too stupid. They often respond with nonsense or fail to follow instructions.

Even like OpenChat-3.5? (Probably the best 7B model out there) Demo: https://openchat.team/ HuggingFace: https://huggingface.co/openchat/openchat_3.5 On the LLM arena (blinded comparisons), it's the third best non-proprietary model: https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboar...

Are there any comparisons with Mistral-instruct? I've yet to see anything under 30B beat it in any way.

Re: Exponentially faster language modelling

#117

Post by voiceblue »

sorokod wrote: post #111
GaggiX wrote: post #110

Earlier quoted context omitted.

Even like OpenChat-3.5? (Probably the best 7B model out there) Demo: https://openchat.team/ HuggingFace: https://huggingface.co/openchat/openchat_3.5 On the LLM arena (blinded comparisons), it's the third best non-proprietary model: https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboar...

What is the sum of odd numbers in this set: 4, 7, 12, 1, 3 The sum of odd numbers in the given set is 4 + 7 + 1 = 12. Therefore, the answer is 12.

You're using an LLM as if you don't understand how or why it works. Math is not what it's for. Here, look at what ChatGPT 4 does (for a case that hasn't been fed to it by a hundred other users yet):

    What is the sum of odd numbers in this set: 12345654321, 123456543212, 123456543213, 12345654324? Output only the sum, no code.

    The sum of the odd numbers in the set {12345654321, 123456543212, 123456543213, 12345654324} is 246913086434.
What does that prove? Only that I am using an LLM poorly and/or do not understand what it is. Using OpenChat-3.5 for what LLMs are actually good at (e.g. asking it for shell commands to perform certain operations, getting some general information about a topic) seems to work surprisingly well for a 7B model.

Re: Exponentially faster language modelling

#118

Post by GaggiX »

moffkalast wrote: post #116
GaggiX wrote: post #110

Earlier quoted context omitted.

Even like OpenChat-3.5? (Probably the best 7B model out there) Demo: https://openchat.team/ HuggingFace: https://huggingface.co/openchat/openchat_3.5 On the LLM arena (blinded comparisons), it's the third best non-proprietary model: https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboar...

Are there any comparisons with Mistral-instruct? I've yet to see anything under 30B beat it in any way.

Mistral-instruct is in the arena, 1008 ELO vs 1070 ELO (OpenChat-3.5)

Re: Exponentially faster language modelling

#119

Post by optimalsolver »

richardw wrote: post #72
WithinReason wrote: post #48

Earlier quoted context omitted.

That's an interesting question. It actually provides a nice way to parallelized training: Pretrain e.g. the first 3 branch levels, which effectively fragments the model into 8 separate parts, which you can continue training across 8 independent servers/nodes with no further communication between the nodes. A central server would run the 1st 3 levels and mark parts of the training set that each node has to train on. M…

Also steps up the economic benefit of, and therefore demand for, botnets. We really need a solution to bad actors controlling vast amounts of compute.

https://en.wikipedia.org/wiki/Hashcash

Re: Exponentially faster language modelling

#120

Post by itissid »

Noob Question: So is the idea to load only specific branches (and by extension log(n) order neurons), right based on the input data. Would this be something that a compiler would do using a JIT trick(because the input needs to be known to get the right branch) to issue a call to the right neurons into memory(SIMD?) to do the Feed Forward?
Post reply on HN