11–16 of 16 posts
Something people don't get about spinlocks is that you're basically saying you own the entire CPU core. In any other situation where the core is shared by multiple processes, it is inherently illogical to use a spinlock.
Unlike GPUs, CPUs aren't designed for massive parallelism. Because of this, batching inference won't necessarily give you a speed boost here. In fact, it can actually slow the process down. Instead, I'd recommend exploring CPU-specific AI optimizations. For instance, leveraging AVX512_BF16 instructions could reduce the inference time by 2x or 3x compared to the results in the article. OpenVINO supports this really we…
https://aws.amazon.com/blogs/compute/accelerate-cpu-based-ai...
Unlike GPUs, CPUs aren't designed for massive parallelism. Because of this, batching inference won't necessarily give you a speed boost here. In fact, it can actually slow the process down. Instead, I'd recommend exploring CPU-specific AI optimizations. For instance, leveraging AVX512_BF16 instructions could reduce the inference time by 2x or 3x compared to the results in the article. OpenVINO supports this really we…
I'm not sure how you manage to be so wrong about something this simple. If you do a single inference at a time, you do GEMV, which spends most of the time loading parameters and then performs one multiplication and one add per parameter. If you do batching, then you get to do GEMM, which means you load the parameter once and perform multiple calculations per parameter. This is faster even for a purely sequential matr…
Unlike GPUs, CPUs aren't designed for massive parallelism. Because of this, batching inference won't necessarily give you a speed boost here. In fact, it can actually slow the process down. Instead, I'd recommend exploring CPU-specific AI optimizations. For instance, leveraging AVX512_BF16 instructions could reduce the inference time by 2x or 3x compared to the results in the article. OpenVINO supports this really we…
Each of its 24 cores can do two 8 wide FMA ops per cycle. Lets say holding a continuous 4 GHz clock speed.
This works out to over 1.5 trillion 32 bit floating point multiplies per cycle.
If you are doing vector matrix multiplies (like in single token no batching). `xW` then each weight loaded sort of gets used in 1 multiplication and 1 addition.
Doing the math you can clearly see even if each weight were just 1 byte you can at most load 130 billion of them in a second from memory.
But in the same timespan you could have done over 1.5 trillion multiplications.
So you are still memory bound.
Spinlocks are basically the heroin of parallel programming. So many people are addicted to them in the pursuit of performance but the truth is that in 99% of all cases, they are a terrible idea. Something people don't get about spinlocks is that you're basically saying you own the entire CPU core. In any other situation where the core is shared by multiple processes, it is inherently illogical to use a spinlock.
That said, I probably wouldn't ship spinlocks in consumer libraries or code I expect to be reused across deployments.