That's within an order of magnitude of RAM. You're getting 16TB of RAM!
I think you severely underestimate RAM bandwidth on modern servers.
OK, maybe not top-of-the-line RAM, but some servers do have RAM that slow!
16 TB and 20,000 IOPS EBS Volumes
11–20 of 38 posts
Re: 16 TB and 20,000 IOPS EBS Volumes
#12Still slow on the scale of things these days. I just provisioned a few servers with over 500,000 IOP/s / 2000MB/s read and write each, 100% SSD with 3-10 year warranties and they use bugger all power. Very low running cost and maintenance overhead and cost less than 8k a unit (1u chassis, redundant power, 32GB RAM, 2x 6 core Xeon v3) and I can guarantee the performance is consistent and there when We need it. I'm all…
Guys this is good information freely given. Please don't down vote frivolously, it discourages this type of sharing.
Re: 16 TB and 20,000 IOPS EBS Volumes
#13At OktaWave[1] 20k IOPS with 2GB bandwidth is only the second tier and you can have up to 200k IOPS with 2GB/s bandwidth. Only size of single volume is max 300GB, but you can raid0 them.
Re: 16 TB and 20,000 IOPS EBS Volumes
#14Earlier quoted context omitted.
I think you severely underestimate RAM bandwidth on modern servers.
OK, maybe not top-of-the-line RAM, but some servers do have RAM that slow!
PCIe or SATA attached SSD random access latency is around 0.2ms. Typical RAM latency is 100ns, maybe 200 in a NUMA cross-node access. That's a 3 orders of magnitude difference. Add another order of magnitude for network-attached SSD.
Bandwidth wise, a single DDR3 channel has around 10 GB/s (and a typical server has 4 to 8 of them). A single half duplex 10GE link (the most you can provision and effectively use on EC2) is 500 MB/s. So, generally 1 to 2 orders of magnitude.
Re: 16 TB and 20,000 IOPS EBS Volumes
#15Earlier quoted context omitted.
Yes we can and do snapshot them, at several levels actually - I don't think that's a particularly hard thing to do so I'm not sure why that's relevant. Yes there is replication both to separate disk arrays AND seperate physical servers with live failover and load balancing - again nothing new here? No we don't send out storage to other countries - in fact that would be illegal, and if we were to do so our clients wou…
You say LUN, are these SAN devices, or is it direct attached storage? Was the replication, load balancing, and snapshotting all something that you set up and manage yourselves? --edit-- Ahh you've been editing your comments so the thread is a bit out of wack! (no problemo) Newegg: http://www.newegg.com/Product/Product.aspx?Item=N82E16820167... ;) (yes its not the same as some of the much higher end stuff). Fair enoug…
Yeah sorry I didn't want it to end up sounding like a threaded argument - and I was sort of brain dumping as I go.
Hardware wise - We use standard servers (super micro), packed with several tiers of SSDs (Intel for the high end, SanDisk for the lower end).
Software wise, again all off the shelf, well understood tools: Debian Linux, DRBD, iSCSI, LACP, LVM, Puppet.
Our compute servers are blades with Debian VMs running Docker containers Of our applications.
Edit: something we've gained greatly from that isn't off the shelf is that we moved to running very modern Linux Kernels - we have CI builds triggered as new stable versions are released and they are stock standard except that we do patch them with GRSecurity and ensure SELinux is enforcing.
All this doesn't cost much time to manage at all - we don't even have a storage admin and to be honest - if we needed one we'd be doing something wrong - apart from physical failure (which is very rare these days) there really isn't anything to do with storage - it's almost boring!
Re: 16 TB and 20,000 IOPS EBS Volumes
#16Earlier quoted context omitted.
You say LUN, are these SAN devices, or is it direct attached storage? Was the replication, load balancing, and snapshotting all something that you set up and manage yourselves? --edit-- Ahh you've been editing your comments so the thread is a bit out of wack! (no problemo) Newegg: http://www.newegg.com/Product/Product.aspx?Item=N82E16820167... ;) (yes its not the same as some of the much higher end stuff). Fair enoug…
Yeah sorry I didn't want it to end up sounding like a threaded argument - and I was sort of brain dumping as I go. Hardware wise - We use standard servers (super micro), packed with several tiers of SSDs (Intel for the high end, SanDisk for the lower end). Software wise, again all off the shelf, well understood tools: Debian Linux, DRBD, iSCSI, LACP, LVM, Puppet. Our compute servers are blades with Debian VMs running…
I actually have to get some sleep now - it's after 1AM here in Aussie, I wanted to stress that I'm absolutely not against using cloud hosted services - just that they're not the answer to all situations and there's a lot to be gained from ensuring you don't get sucked in to too much of the 'Spin' that vendors provide.
Re: 16 TB and 20,000 IOPS EBS Volumes
#17Earlier quoted context omitted.
OK, maybe not top-of-the-line RAM, but some servers do have RAM that slow!
PCIe or SATA attached SSD random access latency is around 0.2ms. Typical RAM latency is 100ns, maybe 200 in a NUMA cross-node access. That's a 3 orders of magnitude difference. Add another order of magnitude for network-attached SSD. Bandwidth wise, a single DDR3 channel has around 10 GB/s (and a typical server has 4 to 8 of them). A single half duplex 10GE link (the most you can provision and effectively use on EC2)…
Ok the latency is high. But I was seeing numbers in the 300,000 IOPS range for ramdisks.
Re: 16 TB and 20,000 IOPS EBS Volumes
#18Earlier quoted context omitted.
Yes we can and do snapshot them, at several levels actually - I don't think that's a particularly hard thing to do so I'm not sure why that's relevant. Yes there is replication both to separate disk arrays AND seperate physical servers with live failover and load balancing - again nothing new here? No we don't send out storage to other countries - in fact that would be illegal, and if we were to do so our clients wou…
You say LUN, are these SAN devices, or is it direct attached storage? Was the replication, load balancing, and snapshotting all something that you set up and manage yourselves? --edit-- Ahh you've been editing your comments so the thread is a bit out of wack! (no problemo) Newegg: http://www.newegg.com/Product/Product.aspx?Item=N82E16820167... ;) (yes its not the same as some of the much higher end stuff). Fair enoug…
FYI - there are laws around where data is allowed to be hosted and what country that parent company is located in. Even if there weren't laws on this - I think it would be pretty irresponsible to trust all your data and servers to one large off shore corporation, especially one that has a fragile political climate (mind you, want countries don't!)
Re: 16 TB and 20,000 IOPS EBS Volumes
#19That's within an order of magnitude of RAM. You're getting 16TB of RAM!
Actually it's considerably slower than the single SSD in the lowest model MacBook Pro.