You could also probably just ask your TAM for this info.
It'd be interesting to compare the results.
How much memory bandwidth do large Amazon instances offer?
11–20 of 50 posts
Re: How much memory bandwidth do large Amazon instances offer?
#12Do you have more information on the results obtained? any information on how to enable NUMA-aware code ? The r6i.metal instance says up to 50 Gbps on Network bandwidth: https://aws.amazon.com/ec2/instance-types/r6i/
50 Gbps is only ~6 GB/s, so you are unlikely to saturate memory bandwidth via network traffic alone
Re: How much memory bandwidth do large Amazon instances offer?
#13>You can double this amount of bandwidth with NUMA-aware code, but it will require further engineering.
So is this benchmark really useful? Seems like it's engineered enough that it probably isn't relevant for most use cases (like running java, go or python services), and not engineered enough so it's not relevant for performant systems like databases. (That said NUMA simply isn't that hard, even with Java which is also NUMA aware).
Re: How much memory bandwidth do large Amazon instances offer?
#14I changed data_volume to 8GB and ran on my 4 year old laptop (Amd Ryzen 4800H + 2 x 8GB 3200MHz DDR4 running NixOS with Linux 6.1.71)
I get around 36 GB/s
1 20.9
2 33.4
3 35.7
4 36.2
5 35.2
6 35.4
7 34.7
8 34.9
9 34.0
10 34.2
11 34.4
12 34.3
13 33.8
14 33.6
15 33.4
16 32.9
Re: How much memory bandwidth do large Amazon instances offer?
#15Running this code on an "on-prem" AMD EPYC 9454P with 12 x 64 GB DIMMs: 1 37.4 2 73.3 3 107.3 4 141.4 5 171.6 6 199.5 7 226.0 8 251.1 9 235.4 10 243.0 11 264.5 12 281.9 13 303.7 14 323.0 15 339.6 16 354.4 17 299.0 18 286.3 19 300.9 20 310.6 21 325.7 22 339.2 23 352.6 24 364.3 25 305.8 26 309.0 27 319.6 28 326.5 29 335.4 30 345.5 31 356.7 32 364.9 And then it settles around there.
[deleted]
Re: How much memory bandwidth do large Amazon instances offer?
#16For the curious like me, here's the script running on consumer gaming Intel hardware (i9 14900K, 2x 48GB DDR5 6600MHz CL32 XMP, Windows 11): 1 27.6 2 50.5 3 62.5 4 68.3 5 75.6 6 82.6 7 87.4 8 90.7 9 93.0 10 94.3 ...repeating I expected a bit more, my guess would have been around 120 GB/s. I've been playing with LLMs and this hardware is about as fast as memory gets on consumer Intel without overclocking. Screenshot s…
[deleted]
Re: How much memory bandwidth do large Amazon instances offer?
#17I changed data_volume to 8GB and ran on my 4 year old laptop (Amd Ryzen 4800H + 2 x 8GB 3200MHz DDR4 running NixOS with Linux 6.1.71) I get around 36 GB/s 1 20.9 2 33.4 3 35.7 4 36.2 5 35.2 6 35.4 7 34.7 8 34.9 9 34.0 10 34.2 11 34.4 12 34.3 13 33.8 14 33.6 15 33.4 16 32.9 I expected 51.2 GB/s (2 x 3200 x 1e6 Transfers/s of 8 bytes each)
[deleted]
Re: How much memory bandwidth do large Amazon instances offer?
#18You could also probably just ask your TAM for this info.
As a former AWS employee — answer will vary wildly!
Re: How much memory bandwidth do large Amazon instances offer?
#197950x3d w/ 128GB at stock timings (~3200MT/s?). Showed a high base but no increase with threads, need to investigate what is happening.
1 54.7
2 50.6
3 49.6
4 48.2
5 47.9
6 47.4
7 47.1
8 46.6
9 46.5
10 46.2
11 46.1
12 45.9
13 45.8
14 45.7
15 45.7
16 45.7
17 45.7
18 45.8
19 45.9
20 45.8
21 45.8
22 45.6
23 45.6
24 45.5
25 45.5
26 45.5
27 45.5
28 45.4
29 45.4
30 45.4
31 45.4
32 45.4Re: How much memory bandwidth do large Amazon instances offer?
#20Do you have more information on the results obtained? any information on how to enable NUMA-aware code ? The r6i.metal instance says up to 50 Gbps on Network bandwidth: https://aws.amazon.com/ec2/instance-types/r6i/
Typically there are NUMA-aware memory allocators. I don't believe glibc is but mimalloc and the non gperftools tcmalloc should be. The Linux kernel is NUMA-aware and will try to avoid shifting work to a different NUMA node but it may depending on what else is happening - it's a generic algorithm trying to balance CPU utilization, latency, and memory bandwidth. If you're using a higher level language like Go or Java then there may be more involved. Java has a NUMA aware allocator while Go's design requires it to also have a NUMA aware scheduler which I don't believe it does.
You could go extreme and start pinning your threads to specific CPUs to tune the code manually instead of relying on the kernel if you know memory bandwidth is extremely important and you won't have much CPU contention to worry about in terms of getting work scheduled in a timely manner.
All that being said, you typically also need to design your application from the ground up to be NUMA aware to take full advantage so that you can set up your allocations to happen on the right zone & whatnot.