Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Bfloat16 support coming to Apple's Metal and PyTorch [video]

developer.apple.com

Post reply on HN
Print view View original

31–40 of 55 posts

Re: Bfloat16 support coming to Apple's Metal and PyTorch [video]

#31

Post by brucethemoose2 »

IOT_Apprentice wrote: post #29
hospitalJail wrote: post #27

Maybe someone can help me understand why people are investing into this. Inhousing typically means falling behind in technology but having lower operating costs. That makes the company win, not the users. If you hinge your career on Apple, they might make your technology obsolete on a dime. Its not the fastest, its not the best, its not the cheapest, its not some combination either. > 'compute per watt' With AI? The…

My question to you is what are you currently using as an alternative for the COU/SOC in your personal & work environments? Intel? AMD Ryzen? Apple has taken their ARM approach and scaled it to all their platforms. Amazon now is on what, Gen 2 or 3 for their graviton platform in AWS. And what OS are you using if you don’t trust Microsoft, Linux or Apple?

CPU arch isnt't even that critical here, as Apple is talking about Metal.

Re: Bfloat16 support coming to Apple's Metal and PyTorch [video]

#32

Post by londons_explore »

kamilner wrote: post #23
londons_explore wrote: post #22

Earlier quoted context omitted.

At just 4 bits, there are only 16 possible numbers. It becomes lookup table territory - and there is no need to have the numbers on your numberline be linearly or exponentially spaced - you can assign them arbitarily. For example, you could have a number system consisting of: (+-) 0.5, 1, 2, 3, 5, 10, 1000, 1000000 - getting some nice accuracy in the middle of the number line where you expect most values to lie, plus…

The more recent 4 bit quantizations are almost along these lines. Q4_1 in ggml for example takes a block of 32 weights and gives each block a scaling factor 'd' and takes the minimum of the weights 'm' to be the quantized '0', so the final weights from a quantized weight 'q' is q * d + m, and taking a relatively small block size makes it more likely that those are all within a reasonable quantization range. Notably,…

Very efficient for storage and memory bandwidth, but such a scheme is a headache for high throughput hardware implementations (at least compared to regular 4 bit math, which can be packed really really densely)

Re: Bfloat16 support coming to Apple's Metal and PyTorch [video]

#33

Post by MuffinFlavored »

brucethemoose2 wrote: post #30
hospitalJail wrote: post #27

Maybe someone can help me understand why people are investing into this. Inhousing typically means falling behind in technology but having lower operating costs. That makes the company win, not the users. If you hinge your career on Apple, they might make your technology obsolete on a dime. Its not the fastest, its not the best, its not the cheapest, its not some combination either. > 'compute per watt' With AI? The…

> Maybe someone can help me understand why people are investing into this. Buying a Mac for running LLMs is kinda like buying a Mac for gaming. Its thoeretically interesting, but I don't think thats a serious driver of Mac sales. But: - Finetuned local LLMs are good for specific niches, like roleplaying, text games, and helper bots for your own pile of data. And they are getting better at other niches like code compl…

> - Finetuned local LLMs are good for specific niches, like roleplaying, text games, and helper bots for your own pile of data.

I can't see how they don't hallucinate/are leagues away from GPT-3.5 let alone GPT-4 level of quality of output. Am I mistaken?

Re: Bfloat16 support coming to Apple's Metal and PyTorch [video]

#34

Post by Me1000 »

hospitalJail wrote: post #27

Maybe someone can help me understand why people are investing into this. Inhousing typically means falling behind in technology but having lower operating costs. That makes the company win, not the users. If you hinge your career on Apple, they might make your technology obsolete on a dime. Its not the fastest, its not the best, its not the cheapest, its not some combination either. > 'compute per watt' With AI? The…

There are a lot of ML applications outside of LLMs. Why would a developer invest in it? Because there are hundreds of millions of iOS devices out there where computer vision, text recognition, etc would be useful features.

Re: Bfloat16 support coming to Apple's Metal and PyTorch [video]

#35

Post by kelnos »

londons_explore wrote: post #17
pklausler wrote: post #16

Earlier quoted context omitted.

Conversions from IEEE-32 to BF16 don't round?

I don't believe the standard defines it. I believe implementations truncate (ie. round towards zero). Remember BF16 was invented specifically to be able to be backwards compatible with existing silicon - and pulling 2 bytes out of 4 is a far cheaper operation than any rounding.

Just to elaborate, as I was confused about this and had to look it up: BF16 is indeed designed to just be a truncated F32: you can grab the top 16 bits of a F32 value and it'll still "make sense": the sign bits are in the same place in both (unsurprisingly), and the exponent part of BF16 and F32 are both 8 bits. In the case of the mantissa, you end up grabbing the top 7 bits of the F32's 23-bit mantissa, so it all works out, as this will "round" the value toward zero.

Re: Bfloat16 support coming to Apple's Metal and PyTorch [video]

#36

Post by my123 »

eoskx wrote: post #2

Somehow missed this from WWDC23, but it looks like Sonoma will add support for bfloat16 with Metal, and there's an active PR to add support with the PyTorch MPS back-end (PR #99272). Since M2 added bfloat16 support at the hardware level, I'm assuming this will only be supported on M2 Macs. That maxed out Mac Studio M2 w/ 192GB of memory now looks more appealing...

bf16 in Metal on macOS 14 is supported on all Macs. Emulated in software transparently.

Re: Bfloat16 support coming to Apple's Metal and PyTorch [video]

#37

Post by woadwarrior01 »

isoprophlex wrote: post #18
londons_explore wrote: post #15

It's a shame that large language models are mostly moving to 4 bit weights for inference, and a bunch of papers have shown promising techniques for training in 4 bit too... Remember that switching from 16 bit to 4 bit lets you have 4x as many weights, 4x as many weights loaded from RAM per second, and ~1/16 of the silicon area for the calculations (a multiplier scales with approximately the number of bits squared). T…

What?! Can you also train with quantization? Incredible! I'd have thought the gradients were way too ugly for any convergence with 4 bits. Any particularly good papers you can recommend me on the topic?

Here's a recent paper on training transformers with 4 bit integer weights.

https://arxiv.org/abs/2306.11987

Re: Bfloat16 support coming to Apple's Metal and PyTorch [video]

#38

Post by maximilianburke »

londons_explore wrote: post #15

It's a shame that large language models are mostly moving to 4 bit weights for inference, and a bunch of papers have shown promising techniques for training in 4 bit too... Remember that switching from 16 bit to 4 bit lets you have 4x as many weights, 4x as many weights loaded from RAM per second, and ~1/16 of the silicon area for the calculations (a multiplier scales with approximately the number of bits squared). T…

Is it possible we will we eventually see 1-bit weights in use?

Re: Bfloat16 support coming to Apple's Metal and PyTorch [video]

#39

Post by brucethemoose2 »

MuffinFlavored wrote: post #33
brucethemoose2 wrote: post #30

Earlier quoted context omitted.

> Maybe someone can help me understand why people are investing into this. Buying a Mac for running LLMs is kinda like buying a Mac for gaming. Its thoeretically interesting, but I don't think thats a serious driver of Mac sales. But: - Finetuned local LLMs are good for specific niches, like roleplaying, text games, and helper bots for your own pile of data. And they are getting better at other niches like code compl…

> - Finetuned local LLMs are good for specific niches, like roleplaying, text games, and helper bots for your own pile of data. I can't see how they don't hallucinate/are leagues away from GPT-3.5 let alone GPT-4 level of quality of output. Am I mistaken?

They are better than GPT 3.5 (which I am generally not impressed with), but not as good as GPT4.

Again, the specialized variants perform very well in their niches.

Re: Bfloat16 support coming to Apple's Metal and PyTorch [video]

#40

Post by brucethemoose2 »

maximilianburke wrote: post #38
londons_explore wrote: post #15

It's a shame that large language models are mostly moving to 4 bit weights for inference, and a bunch of papers have shown promising techniques for training in 4 bit too... Remember that switching from 16 bit to 4 bit lets you have 4x as many weights, 4x as many weights loaded from RAM per second, and ~1/16 of the silicon area for the calculations (a multiplier scales with approximately the number of bits squared). T…

Is it possible we will we eventually see 1-bit weights in use?

There are already papers on it, and there is 2-bit quant in llama.cpp.

But it seems to be past the point of diminishing returns, where you mind as well use a model with fewer parameters... For now.

There was another scheme in a paper where the "sparse" majority of the model was highly quantized, while the "dense" part was left in FP16, with good results.

Post reply on HN