Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Bfloat16 support coming to Apple's Metal and PyTorch [video]

developer.apple.com

Post reply on HN
Print view View original

21–30 of 55 posts

Re: Bfloat16 support coming to Apple's Metal and PyTorch [video]

#21

Post by londons_explore »

sbierwagen wrote: post #19
isoprophlex wrote: post #18

Earlier quoted context omitted.

What?! Can you also train with quantization? Incredible! I'd have thought the gradients were way too ugly for any convergence with 4 bits. Any particularly good papers you can recommend me on the topic?

A group at IBM has been working on minifloat training for a while. Here's a paper from 2020 on FP4 training: https://papers.nips.cc/paper/2020/file/13b919438259814cd5be8...

Their best performing 4-bit number format uses 1 sign bit, 3 exponent bits, and no mantissa bits!

Ie. All weights, activations and gradients become powers of two! Which means all multiplications become simple bit shifts. That really changes mathematics and silicon design.

Re: Bfloat16 support coming to Apple's Metal and PyTorch [video]

#22

Post by londons_explore »

sbierwagen wrote: post #19
isoprophlex wrote: post #18

Earlier quoted context omitted.

What?! Can you also train with quantization? Incredible! I'd have thought the gradients were way too ugly for any convergence with 4 bits. Any particularly good papers you can recommend me on the topic?

A group at IBM has been working on minifloat training for a while. Here's a paper from 2020 on FP4 training: https://papers.nips.cc/paper/2020/file/13b919438259814cd5be8...

At just 4 bits, there are only 16 possible numbers. It becomes lookup table territory - and there is no need to have the numbers on your numberline be linearly or exponentially spaced - you can assign them arbitarily. For example, you could have a number system consisting of: (+-) 0.5, 1, 2, 3, 5, 10, 1000, 1000000 - getting some nice accuracy in the middle of the number line where you expect most values to lie, plus some extreme values so convergence doesn't take forever if some big activation/gradient needs to be propagated.

Re: Bfloat16 support coming to Apple's Metal and PyTorch [video]

#23

Post by kamilner »

londons_explore wrote: post #22
sbierwagen wrote: post #19

Earlier quoted context omitted.

A group at IBM has been working on minifloat training for a while. Here's a paper from 2020 on FP4 training: https://papers.nips.cc/paper/2020/file/13b919438259814cd5be8...

At just 4 bits, there are only 16 possible numbers. It becomes lookup table territory - and there is no need to have the numbers on your numberline be linearly or exponentially spaced - you can assign them arbitarily. For example, you could have a number system consisting of: (+-) 0.5, 1, 2, 3, 5, 10, 1000, 1000000 - getting some nice accuracy in the middle of the number line where you expect most values to lie, plus…

The more recent 4 bit quantizations are almost along these lines. Q4_1 in ggml for example takes a block of 32 weights and gives each block a scaling factor 'd' and takes the minimum of the weights 'm' to be the quantized '0', so the final weights from a quantized weight 'q' is q * d + m, and taking a relatively small block size makes it more likely that those are all within a reasonable quantization range. Notably, d and m can be stored with more accuracy without sacrificing too much space, since the overhead is divided by 32. Q4_k goes a bit further, and takes 'superblocks' of 8 blocks, and applies another scaling factor 'd_s' and minimum 'm_s' to that, so the final weight is (q * d + m) * d_s + m_s, and the additional factors are stored as 6 bits instead of 4.

In practice this seems to get very good results, while being cheap to implement and relatively space efficient, Q4_K for example works out to 4.5 bits per weight instead of 4. The PR adding it has more details: https://github.com/ggerganov/llama.cpp/pull/1684

Re: Bfloat16 support coming to Apple's Metal and PyTorch [video]

#24

Post by brucethemoose2 »

londons_explore wrote: post #15

It's a shame that large language models are mostly moving to 4 bit weights for inference, and a bunch of papers have shown promising techniques for training in 4 bit too... Remember that switching from 16 bit to 4 bit lets you have 4x as many weights, 4x as many weights loaded from RAM per second, and ~1/16 of the silicon area for the calculations (a multiplier scales with approximately the number of bits squared). T…

There is some overhead from the quantization, and right now the operations themself are sometimes done at higher precision than the weights in RAM.

And widespread hardware 4 bit will take some time. If the HW makers started designing 4 bit silicon in 2022, then we are still years away.

Re: Bfloat16 support coming to Apple's Metal and PyTorch [video]

#25

Post by bravura »

minimaxir wrote: post #3

I'm still confused by the proliferation of bf16. Although it certainly doesn't hurt compared to fp16, in my testing even with A100 GPUs optimized for it, both training speed and inference quality are the same between bf16 and fp16.

Despite the other answers, I will tell you the grim truth: Your mileage might vary.

It's an empirical question and depends upon the nature of your problem and data. You should try all three fp32, fp16, and bf16 as part our model selection / hyperparameter tuning.

For example, in audio generative models (where typical output is 16-bit), I've sometimes found that fp16 and bf16 just don't produce good output as fp32 weights.

Re: Bfloat16 support coming to Apple's Metal and PyTorch [video]

#26

Post by BooneJS »

sillysaurusx wrote: post #20
londons_explore wrote: post #13

Earlier quoted context omitted.

Luckily BF16 is just a truncated FP32. That means that the hardware can do BF16, just you don't get any performance benefit compared to FP32 (and depending on the hardware design, you might also have to space the data 4 bytes apart rather than 2), so you lose the memory bandwidth and RAM usage benefits too.

At that point it’d be better to do everything in fp32. The hardware can’t do bf16 in the way you’re saying; the conversions would consume all your time.

Compute in F32, but then round and pack a pair of BF16 into 4 bytes.

Re: Bfloat16 support coming to Apple's Metal and PyTorch [video]

#27

Post by hospitalJail »

Maybe someone can help me understand why people are investing into this.

Inhousing typically means falling behind in technology but having lower operating costs. That makes the company win, not the users.

If you hinge your career on Apple, they might make your technology obsolete on a dime.

Its not the fastest, its not the best, its not the cheapest, its not some combination either.

> 'compute per watt'

With AI? The local LLM models are near useless already. There will be a time to cut down on power, but from what I've read, there is currently ~no value even with a 4090 with 512 RAM.

I suggest avoiding Windows/M$, I am annoyed with Linux bugs, and google cannot be trusted. But all of that could be said about Apple as well.

I just don't see a future with Apple hardware, it gives me some serious Nintendo vibes where they are going to be some quirky niche that is just enough for marketers to sell it. Compute per watt seems like a wiimote that no one asked for, but suddenly claim is ultra important.

Maybe someone can change my view. I don't see who buys this when they are educated on the possible options.

Re: Bfloat16 support coming to Apple's Metal and PyTorch [video]

#28

Post by voz_ »

redox99 wrote: post #8
minimaxir wrote: post #3

I'm still confused by the proliferation of bf16. Although it certainly doesn't hurt compared to fp16, in my testing even with A100 GPUs optimized for it, both training speed and inference quality are the same between bf16 and fp16.

Sometimes during training, fp16 will cause networks that would converge on fp32, to explode to Infs or NaNs with fp16, because of the limited range. bf16 generally speaking fixes that. It's true also that fp16 is often manageable with enough batch/layer norm and gradient clipping.

Yea, I spent a few months comparing the two, and empirically i had a lot more issues with various normalized entropy problems (explosion, not converging, converging slower) with fp16 than with bf16.

The transfer pipeline I wrote for fp32->fp16 also took a lot more work than fp32->bf16

Re: Bfloat16 support coming to Apple's Metal and PyTorch [video]

#29

Post by IOT_Apprentice »

hospitalJail wrote: post #27

Maybe someone can help me understand why people are investing into this. Inhousing typically means falling behind in technology but having lower operating costs. That makes the company win, not the users. If you hinge your career on Apple, they might make your technology obsolete on a dime. Its not the fastest, its not the best, its not the cheapest, its not some combination either. > 'compute per watt' With AI? The…

My question to you is what are you currently using as an alternative for the COU/SOC in your personal & work environments?

Intel? AMD Ryzen?

Apple has taken their ARM approach and scaled it to all their platforms.

Amazon now is on what, Gen 2 or 3 for their graviton platform in AWS.

And what OS are you using if you don’t trust Microsoft, Linux or Apple?

Re: Bfloat16 support coming to Apple's Metal and PyTorch [video]

#30

Post by brucethemoose2 »

hospitalJail wrote: post #27

Maybe someone can help me understand why people are investing into this. Inhousing typically means falling behind in technology but having lower operating costs. That makes the company win, not the users. If you hinge your career on Apple, they might make your technology obsolete on a dime. Its not the fastest, its not the best, its not the cheapest, its not some combination either. > 'compute per watt' With AI? The…

> Maybe someone can help me understand why people are investing into this.

Buying a Mac for running LLMs is kinda like buying a Mac for gaming. Its thoeretically interesting, but I don't think thats a serious driver of Mac sales.

But:

- Finetuned local LLMs are good for specific niches, like roleplaying, text games, and helper bots for your own pile of data. And they are getting better at other niches like code completion for specific languages, or summarization.

- Remember that a huge selling point for Macs is iPhone/iPad development. The market for AI App Store apps is not small.This is also a reason to believe there will be some stability with the ML support.

Post reply on HN