Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Bfloat16 support coming to Apple's Metal and PyTorch [video]

developer.apple.com

Post reply on HN
Print view View original

41–50 of 55 posts

Re: Bfloat16 support coming to Apple's Metal and PyTorch [video]

#41

Post by Dylan16807 »

londons_explore wrote: post #21
sbierwagen wrote: post #19

Earlier quoted context omitted.

A group at IBM has been working on minifloat training for a while. Here's a paper from 2020 on FP4 training: https://papers.nips.cc/paper/2020/file/13b919438259814cd5be8...

Their best performing 4-bit number format uses 1 sign bit, 3 exponent bits, and no mantissa bits! Ie. All weights, activations and gradients become powers of two! Which means all multiplications become simple bit shifts. That really changes mathematics and silicon design.

Does it really make much of a difference?

You're usually feeding a ton of multiplies into an accumulator. You can handle one or two mantissa bits as the same bit shifting except that it outputs two or three numbers to accumulate. And accumulators are very easy to scale.

Also in the extreme I've seen powers of 4 get used.

Re: Bfloat16 support coming to Apple's Metal and PyTorch [video]

#42

Post by [deleted] »

brucethemoose2 wrote: post #30
hospitalJail wrote: post #27

Maybe someone can help me understand why people are investing into this. Inhousing typically means falling behind in technology but having lower operating costs. That makes the company win, not the users. If you hinge your career on Apple, they might make your technology obsolete on a dime. Its not the fastest, its not the best, its not the cheapest, its not some combination either. > 'compute per watt' With AI? The…

> Maybe someone can help me understand why people are investing into this. Buying a Mac for running LLMs is kinda like buying a Mac for gaming. Its thoeretically interesting, but I don't think thats a serious driver of Mac sales. But: - Finetuned local LLMs are good for specific niches, like roleplaying, text games, and helper bots for your own pile of data. And they are getting better at other niches like code compl…

[deleted]

Re: Bfloat16 support coming to Apple's Metal and PyTorch [video]

#43

Post by Geee »

hospitalJail wrote: post #27

Maybe someone can help me understand why people are investing into this. Inhousing typically means falling behind in technology but having lower operating costs. That makes the company win, not the users. If you hinge your career on Apple, they might make your technology obsolete on a dime. Its not the fastest, its not the best, its not the cheapest, its not some combination either. > 'compute per watt' With AI? The…

Desktop computers are heat-limited. We could have much faster computers if we found a way to cool them down. Thus, compute per watt is the ultimate metric to optimize for. If your cooling capacity is 500W, then obviously you'll want to fit as much compute in that as possible.

Mobile devices are energy-limited. You'll want to do as much compute as possible on a limited battery.

Re: Bfloat16 support coming to Apple's Metal and PyTorch [video]

#44

Post by LoganDark »

kamilner wrote: post #23
londons_explore wrote: post #22

Earlier quoted context omitted.

At just 4 bits, there are only 16 possible numbers. It becomes lookup table territory - and there is no need to have the numbers on your numberline be linearly or exponentially spaced - you can assign them arbitarily. For example, you could have a number system consisting of: (+-) 0.5, 1, 2, 3, 5, 10, 1000, 1000000 - getting some nice accuracy in the middle of the number line where you expect most values to lie, plus…

The more recent 4 bit quantizations are almost along these lines. Q4_1 in ggml for example takes a block of 32 weights and gives each block a scaling factor 'd' and takes the minimum of the weights 'm' to be the quantized '0', so the final weights from a quantized weight 'q' is q * d + m, and taking a relatively small block size makes it more likely that those are all within a reasonable quantization range. Notably,…

This is an excellent explanation, thank you!!

Re: Bfloat16 support coming to Apple's Metal and PyTorch [video]

#45

Post by LoganDark »

my123 wrote: post #36
eoskx wrote: post #2

Somehow missed this from WWDC23, but it looks like Sonoma will add support for bfloat16 with Metal, and there's an active PR to add support with the PyTorch MPS back-end (PR #99272). Since M2 added bfloat16 support at the hardware level, I'm assuming this will only be supported on M2 Macs. That maxed out Mac Studio M2 w/ 192GB of memory now looks more appealing...

bf16 in Metal on macOS 14 is supported on all Macs. Emulated in software transparently.

Yeah, Metal is pretty great because it runs the same on all Macs. Apple is really really good at this.

Re: Bfloat16 support coming to Apple's Metal and PyTorch [video]

#46

Post by pclmulqdq »

londons_explore wrote: post #17
pklausler wrote: post #16

Earlier quoted context omitted.

Conversions from IEEE-32 to BF16 don't round?

I don't believe the standard defines it. I believe implementations truncate (ie. round towards zero). Remember BF16 was invented specifically to be able to be backwards compatible with existing silicon - and pulling 2 bytes out of 4 is a far cheaper operation than any rounding.

There's no standardized definition of BF16.

Re: Bfloat16 support coming to Apple's Metal and PyTorch [video]

#48

Post by stephencanon »

londons_explore wrote: post #13
dlewis1788 wrote: post #7

Confirmed Apple M1 lacks bfloat16 support completely - M1: hw.optional.arm.FEAT_BF16: 0 vs M2: hw.optional.arm.FEAT_BF16: 1

Luckily BF16 is just a truncated FP32. That means that the hardware can do BF16, just you don't get any performance benefit compared to FP32 (and depending on the hardware design, you might also have to space the data 4 bytes apart rather than 2), so you lose the memory bandwidth and RAM usage benefits too.

You still get a perf benefit from half the memory traffic and keeping twice as much data in caches, since you can do the expansion to f32 when loading into registers.

Re: Bfloat16 support coming to Apple's Metal and PyTorch [video]

#50

Post by brrrrrm »

sillysaurusx wrote: post #20
londons_explore wrote: post #13

Earlier quoted context omitted.

Luckily BF16 is just a truncated FP32. That means that the hardware can do BF16, just you don't get any performance benefit compared to FP32 (and depending on the hardware design, you might also have to space the data 4 bytes apart rather than 2), so you lose the memory bandwidth and RAM usage benefits too.

At that point it’d be better to do everything in fp32. The hardware can’t do bf16 in the way you’re saying; the conversions would consume all your time.

The conversions are just a mask and shift? Super cheap
Post reply on HN