Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How much memory bandwidth do large Amazon instances offer?

lemire.me

Post reply on HN
Print view View original

1–10 of 50 posts

Re: How much memory bandwidth do large Amazon instances offer?

#2

Post by jakebsky »

Running this code on an "on-prem" AMD EPYC 9454P with 12 x 64 GB DIMMs:

  1 37.4
  2 73.3
  3 107.3
  4 141.4
  5 171.6
  6 199.5
  7 226.0
  8 251.1
  9 235.4
  10 243.0
  11 264.5
  12 281.9
  13 303.7
  14 323.0
  15 339.6
  16 354.4
  17 299.0
  18 286.3
  19 300.9
  20 310.6
  21 325.7
  22 339.2
  23 352.6
  24 364.3
  25 305.8
  26 309.0
  27 319.6
  28 326.5
  29 335.4
  30 345.5
  31 356.7
  32 364.9
And then it settles around there.

Re: How much memory bandwidth do large Amazon instances offer?

#4

Post by bgirard »

jakebsky wrote: post #2

Running this code on an "on-prem" AMD EPYC 9454P with 12 x 64 GB DIMMs: 1 37.4 2 73.3 3 107.3 4 141.4 5 171.6 6 199.5 7 226.0 8 251.1 9 235.4 10 243.0 11 264.5 12 281.9 13 303.7 14 323.0 15 339.6 16 354.4 17 299.0 18 286.3 19 300.9 20 310.6 21 325.7 22 339.2 23 352.6 24 364.3 25 305.8 26 309.0 27 319.6 28 326.5 29 335.4 30 345.5 31 356.7 32 364.9 And then it settles around there.

What are the units? MB/s, GB/s

Re: How much memory bandwidth do large Amazon instances offer?

#5

Post by deadmutex »

bgirard wrote: post #4
jakebsky wrote: post #2

Running this code on an "on-prem" AMD EPYC 9454P with 12 x 64 GB DIMMs: 1 37.4 2 73.3 3 107.3 4 141.4 5 171.6 6 199.5 7 226.0 8 251.1 9 235.4 10 243.0 11 264.5 12 281.9 13 303.7 14 323.0 15 339.6 16 354.4 17 299.0 18 286.3 19 300.9 20 310.6 21 325.7 22 339.2 23 352.6 24 364.3 25 305.8 26 309.0 27 319.6 28 326.5 29 335.4 30 345.5 31 356.7 32 364.9 And then it settles around there.

What are the units? MB/s, GB/s

looks like GB/s

Re: How much memory bandwidth do large Amazon instances offer?

#7

Post by buzzdenver »

jakebsky wrote: post #2

Running this code on an "on-prem" AMD EPYC 9454P with 12 x 64 GB DIMMs: 1 37.4 2 73.3 3 107.3 4 141.4 5 171.6 6 199.5 7 226.0 8 251.1 9 235.4 10 243.0 11 264.5 12 281.9 13 303.7 14 323.0 15 339.6 16 354.4 17 299.0 18 286.3 19 300.9 20 310.6 21 325.7 22 339.2 23 352.6 24 364.3 25 305.8 26 309.0 27 319.6 28 326.5 29 335.4 30 345.5 31 356.7 32 364.9 And then it settles around there.

So around 3x of what AWS has? I guess this is more about what motherboard they have, and not an on-prem vs cloud difference.

Re: How much memory bandwidth do large Amazon instances offer?

#8

Post by EarthLaunch »

For the curious like me, here's the script running on consumer gaming Intel hardware (i9 14900K, 2x 48GB DDR5 6600MHz CL32 XMP, Windows 11):

  1 27.6
  2 50.5
  3 62.5
  4 68.3
  5 75.6
  6 82.6
  7 87.4
  8 90.7
  9 93.0
  10 94.3
  ...repeating
I expected a bit more, my guess would have been around 120 GB/s. I've been playing with LLMs and this hardware is about as fast as memory gets on consumer Intel without overclocking.

Screenshot showing usage: https://i.imgur.com/okdqgG9.png

Re: How much memory bandwidth do large Amazon instances offer?

#9

Post by champtar »

buzzdenver wrote: post #7
jakebsky wrote: post #2

Running this code on an "on-prem" AMD EPYC 9454P with 12 x 64 GB DIMMs: 1 37.4 2 73.3 3 107.3 4 141.4 5 171.6 6 199.5 7 226.0 8 251.1 9 235.4 10 243.0 11 264.5 12 281.9 13 303.7 14 323.0 15 339.6 16 354.4 17 299.0 18 286.3 19 300.9 20 310.6 21 325.7 22 339.2 23 352.6 24 364.3 25 305.8 26 309.0 27 319.6 28 326.5 29 335.4 30 345.5 31 356.7 32 364.9 And then it settles around there.

So around 3x of what AWS has? I guess this is more about what motherboard they have, and not an on-prem vs cloud difference.

Number of DIMM is important, I know the standard BOM at my company has way more memory that we use just to have more bandwidth.

Re: How much memory bandwidth do large Amazon instances offer?

#10

Post by adrian_b »

buzzdenver wrote: post #7
jakebsky wrote: post #2

Running this code on an "on-prem" AMD EPYC 9454P with 12 x 64 GB DIMMs: 1 37.4 2 73.3 3 107.3 4 141.4 5 171.6 6 199.5 7 226.0 8 251.1 9 235.4 10 243.0 11 264.5 12 281.9 13 303.7 14 323.0 15 339.6 16 354.4 17 299.0 18 286.3 19 300.9 20 310.6 21 325.7 22 339.2 23 352.6 24 364.3 25 305.8 26 309.0 27 319.6 28 326.5 29 335.4 30 345.5 31 356.7 32 364.9 And then it settles around there.

So around 3x of what AWS has? I guess this is more about what motherboard they have, and not an on-prem vs cloud difference.

1.5 times greater speed comes from 768-bit vs. 512-bit width of the memory interface.

Another 1.5 times greater speed comes from DDR5-4800 vs. DDR4-3200.

The rest may be from virtualization and other overheads.

Post reply on HN