Faster embeddings: how we rebuilt the ONNX path in Manticore
manticoresearch.com
Faster embeddings: how we rebuilt the ONNX path in Manticore
1–10 of 16 posts
Re: Faster embeddings: how we rebuilt the ONNX path in Manticore
#2ONNX is my first suggestion to people looking for speed gains on CPU
Re: Faster embeddings: how we rebuilt the ONNX path in Manticore
#3We really need a replacement for all-MiniLM-L12-v2 that can create more robust embeddings with the same compute.
You can technically do Q4 quantization for larger embedding models but I am not sure if that plays nice with ONNX.
Re: Faster embeddings: how we rebuilt the ONNX path in Manticore
#4Unlike GPUs, CPUs aren't designed for massive parallelism. Because of this, batching inference won't necessarily give you a speed boost here. In fact, it can actually slow the process down.
Instead, I'd recommend exploring CPU-specific AI optimizations. For instance, leveraging AVX512_BF16 instructions could reduce the inference time by 2x or 3x compared to the results in the article. OpenVINO supports this really well on Intel CPUs, and converting an ONNX model to OpenVINO is straightforward.
Re: Faster embeddings: how we rebuilt the ONNX path in Manticore
#5Unlike GPUs, CPUs aren't designed for massive parallelism. Because of this, batching inference won't necessarily give you a speed boost here. In fact, it can actually slow the process down. Instead, I'd recommend exploring CPU-specific AI optimizations. For instance, leveraging AVX512_BF16 instructions could reduce the inference time by 2x or 3x compared to the results in the article. OpenVINO supports this really we…
+1 for OpenVINO, we utilise it for our model. It's quite amazing the inference speed you can get from CPUs that most people would assume are running on a GPU.
Re: Faster embeddings: how we rebuilt the ONNX path in Manticore
#6We really need a replacement for all-MiniLM-L12-v2 that can create more robust embeddings with the same compute. You can technically do Q4 quantization for larger embedding models but I am not sure if that plays nice with ONNX.
it's a pain in the ass to do properly.
what we really need it something like auto-round for ONNX
Re: Faster embeddings: how we rebuilt the ONNX path in Manticore
#7Unlike GPUs, CPUs aren't designed for massive parallelism. Because of this, batching inference won't necessarily give you a speed boost here. In fact, it can actually slow the process down. Instead, I'd recommend exploring CPU-specific AI optimizations. For instance, leveraging AVX512_BF16 instructions could reduce the inference time by 2x or 3x compared to the results in the article. OpenVINO supports this really we…
ONNX has AVX512 CPU kernels too, and openvino uses ONNX internally (and ONNX supports openvino backend)
Re: Faster embeddings: how we rebuilt the ONNX path in Manticore
#8We really need a replacement for all-MiniLM-L12-v2 that can create more robust embeddings with the same compute. You can technically do Q4 quantization for larger embedding models but I am not sure if that plays nice with ONNX.
[deleted]
Re: Faster embeddings: how we rebuilt the ONNX path in Manticore
#9Unlike GPUs, CPUs aren't designed for massive parallelism. Because of this, batching inference won't necessarily give you a speed boost here. In fact, it can actually slow the process down. Instead, I'd recommend exploring CPU-specific AI optimizations. For instance, leveraging AVX512_BF16 instructions could reduce the inference time by 2x or 3x compared to the results in the article. OpenVINO supports this really we…
ONNX has AVX512 CPU kernels too, and openvino uses ONNX internally (and ONNX supports openvino backend)
> openvino uses ONNX internally
OpenVINO only uses ONNX to parse the model, not to execute it. It runs computations through its own highly optimized inference engine specifically designed for Intel hardware. It doesn't rely on the ONNX engine at all, and it will even automatically convert eligible model weights to BF16 for you
Re: Faster embeddings: how we rebuilt the ONNX path in Manticore
#10Unlike GPUs, CPUs aren't designed for massive parallelism. Because of this, batching inference won't necessarily give you a speed boost here. In fact, it can actually slow the process down. Instead, I'd recommend exploring CPU-specific AI optimizations. For instance, leveraging AVX512_BF16 instructions could reduce the inference time by 2x or 3x compared to the results in the article. OpenVINO supports this really we…
I'm not sure how you manage to be so wrong about something this simple.
If you do a single inference at a time, you do GEMV, which spends most of the time loading parameters and then performs one multiplication and one add per parameter.
If you do batching, then you get to do GEMM, which means you load the parameter once and perform multiple calculations per parameter. This is faster even for a purely sequential matrix multiplication implementation. CPUs tend to have both SIMD and multiple cores these days. This means that your computational resources exceed the available memory bandwidth by far.
What you suggested in the second "paragraph" is just letting someone else do the batching but with a lower precision data type. You're starting to contradict your first point.