Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Bfloat16 support coming to Apple's Metal and PyTorch [video]

developer.apple.com

Post reply on HN
Print view View original

51–55 of 55 posts

Re: Bfloat16 support coming to Apple's Metal and PyTorch [video]

#51

Post by MrBuddyCasino »

londons_explore wrote: post #15

It's a shame that large language models are mostly moving to 4 bit weights for inference, and a bunch of papers have shown promising techniques for training in 4 bit too... Remember that switching from 16 bit to 4 bit lets you have 4x as many weights, 4x as many weights loaded from RAM per second, and ~1/16 of the silicon area for the calculations (a multiplier scales with approximately the number of bits squared). T…

I dimly remember reading that the mathematical compute-per-density optimum is around 3.x bits in a „brain like structure“, I don’t remember any details though or the precise context. Does this ring a bell with anyone?

Re: Bfloat16 support coming to Apple's Metal and PyTorch [video]

#52

Post by vmirnv »

kamilner wrote: post #23
londons_explore wrote: post #22

Earlier quoted context omitted.

At just 4 bits, there are only 16 possible numbers. It becomes lookup table territory - and there is no need to have the numbers on your numberline be linearly or exponentially spaced - you can assign them arbitarily. For example, you could have a number system consisting of: (+-) 0.5, 1, 2, 3, 5, 10, 1000, 1000000 - getting some nice accuracy in the middle of the number line where you expect most values to lie, plus…

The more recent 4 bit quantizations are almost along these lines. Q4_1 in ggml for example takes a block of 32 weights and gives each block a scaling factor 'd' and takes the minimum of the weights 'm' to be the quantized '0', so the final weights from a quantized weight 'q' is q * d + m, and taking a relatively small block size makes it more likely that those are all within a reasonable quantization range. Notably,…

Also I would highly recommend Q5_K_M for both 7B and 13B models.

It has the best balance between quality and weight of the model and almost indistinguishable from original f16: https://www.reddit.com/r/LocalLLaMA/comments/142q5k5/updated...

Re: Bfloat16 support coming to Apple's Metal and PyTorch [video]

#53

Post by gmerc »

MuffinFlavored wrote: post #33
brucethemoose2 wrote: post #30

Earlier quoted context omitted.

> Maybe someone can help me understand why people are investing into this. Buying a Mac for running LLMs is kinda like buying a Mac for gaming. Its thoeretically interesting, but I don't think thats a serious driver of Mac sales. But: - Finetuned local LLMs are good for specific niches, like roleplaying, text games, and helper bots for your own pile of data. And they are getting better at other niches like code compl…

> - Finetuned local LLMs are good for specific niches, like roleplaying, text games, and helper bots for your own pile of data. I can't see how they don't hallucinate/are leagues away from GPT-3.5 let alone GPT-4 level of quality of output. Am I mistaken?

You seem to assume hallucinations are a fatal flaw. You give it a document to summarize and see how often it hallucinates. Very little. Human performance.

Now often does a human make random shit up about general knowledge questions?

Re: Bfloat16 support coming to Apple's Metal and PyTorch [video]

#54

Post by touisteur »

maximilianburke wrote: post #38
londons_explore wrote: post #15

It's a shame that large language models are mostly moving to 4 bit weights for inference, and a bunch of papers have shown promising techniques for training in 4 bit too... Remember that switching from 16 bit to 4 bit lets you have 4x as many weights, 4x as many weights loaded from RAM per second, and ~1/16 of the silicon area for the calculations (a multiplier scales with approximately the number of bits squared). T…

Is it possible we will we eventually see 1-bit weights in use?

For some time I played with Brevitas and Xilinx's FINN, you could quantize like crazy. I haven't looked since transformers took over the AI world where they were.

Re: Bfloat16 support coming to Apple's Metal and PyTorch [video]

#55

Post by astrange »

MuffinFlavored wrote: post #33
brucethemoose2 wrote: post #30

Earlier quoted context omitted.

> Maybe someone can help me understand why people are investing into this. Buying a Mac for running LLMs is kinda like buying a Mac for gaming. Its thoeretically interesting, but I don't think thats a serious driver of Mac sales. But: - Finetuned local LLMs are good for specific niches, like roleplaying, text games, and helper bots for your own pile of data. And they are getting better at other niches like code compl…

> - Finetuned local LLMs are good for specific niches, like roleplaying, text games, and helper bots for your own pile of data. I can't see how they don't hallucinate/are leagues away from GPT-3.5 let alone GPT-4 level of quality of output. Am I mistaken?

Hallucinations are exactly what you want in a gaming model. That's another way of saying "creativity".
Post reply on HN