41–50 of 72 posts
Would love to know if anyone else had data on: * Impact on M5/C5 instances over similar time period, any difference with the Nitro hypervisor? * Were Dedicated instances ( https://aws.amazon.com/ec2/purchasing-options/dedicated-inst... ) patched as well? * Other examples of software that adapted batching performance automatically with increase in call latency.
It would be nice if AWS could write something official about what they are doing. I've been noticing major performance changes in our instances and have no idea if it is related to Meltdown or something else. Google released a blog post specifically on performance: https://blog.google/topics/google-cloud/protecting-our-googl... It would be nice to have similar transparency from AWS.
Google likely only wrote a blog post about it because they were able to find a way to brag that there was effectively no performance hit. They have never written any blog posts IIRC explaining bad/unpredictable performance on GCE
Think the constant ragging on Google that appears on HN is becoming a little too much.
Would love to know if anyone else had data on: * Impact on M5/C5 instances over similar time period, any difference with the Nitro hypervisor? * Were Dedicated instances ( https://aws.amazon.com/ec2/purchasing-options/dedicated-inst... ) patched as well? * Other examples of software that adapted batching performance automatically with increase in call latency.
We had a lot of m5 and c5 servers randomly die. It was as if someone was running chaos monkey from Netflix in our VPCs...
Earlier quoted context omitted.
It may impact the AWS control plane and amazon.com, but as far as AWS services go, it just means customers will be paying for more instances.
But for VM, do customers pay by CPU usage or runtime? I am sure the Netflix of this world optimise their CPU usage but I also suspect the majority of VMs are mostly idle or have little traffic as their task are either intermittent or are sized for peak usage.
Conceptually it is "pay as you use" by CPU usage, but just rounded into buckets by instance tier.
Of course, there's a lot of underutilization within each bucket, because the granularity isn't per 1% used, but (more or less) per 100% used (aka each core). And also, most applications can't switch instance tiers easily to adapt to demand (though some certainly can).
Earlier quoted context omitted.
That's not the only issue; there's also a lot of compliance issues that having a hosting company can take care of. There's whole sections of PCI and HIPPA compliance that you can just write off as "not our problem, talk to AWS".
PCI compliance had little to do with where you're hosted, but how you're hosted and what you do with data. AWS is not a magical PCI compliance button.
Earlier quoted context omitted.
But for VM, do customers pay by CPU usage or runtime? I am sure the Netflix of this world optimise their CPU usage but I also suspect the majority of VMs are mostly idle or have little traffic as their task are either intermittent or are sized for peak usage.
Technically, you pay for both time (hours on) and CPU Usage (instance tier). Its not like different instance tiers (at least in the same class) use fundamentally more or less powerful processors. They all use the same processors, you just get more or less of it depending on what you pay. Conceptually it is "pay as you use" by CPU usage, but just rounded into buckets by instance tier. Of course, there's a lot of under…
There is a variety of CPUs. You can "cat /proc/cpuinfo" to see what you got.
>>> most applications can't switch instance tiers easily to adapt to demand (though some certainly can).
Most applications can and do change. Changing the instance type is just a reboot of the machine.
Earlier quoted context omitted.
You forgot to type 5 or 6 zeros there.
https://www.kimsufi.com/us/en/ Granted, that's the bottom of the barrel (single disk, no IPKVM etc.), but $100 keeps you running for over a year. Better servers are easily available as well, usually a couple of times cheaper than AWS. Is this a US thing? Based on HN only, I'd never know there's anything between the public cloud and racks of own hardware that you have to wire up and maintain. I have a bunch of quad co…
Saying that they should run on kumsufi is like explaining to a wholesale company that they should use motorbikes instead of trucks, because motorbikes are cheaper.
That's one interesting aspect of these issues and mitigations is that performance really depends on the workload. Just because Google saw little performance impact on their servers, doesn't mean your application won't see. Or because someone said their CPU usage went up 2x doesn't mean it will go up for you. On an unrelated note, kind of wish Meltdown had been discovered and exposed separately from Spectre. Intel has…
Just a nit, we said [1] that not only are our own applications in production doing fine (even against Variant 2) but also we haven’t been inundated with support calls over the last few months while mitigations were silently rolled out at the host kernel and hypervisor layer. So this class of “Hey, my instances are suddenly way slower, I didn’t do anything” isn’t happening on GCE.
That does not mean that if you perform guest OS updates, don’t have PCID enabled, etc. that you won’t see a degradation. We certainly haven’t tried all permutations of all guests, and the kernel patches are still improving. That’s why we’re actively trying to get everyone to rally behind retpoline, KPTI (with PCID enabled), and so on.
[1] https://www.blog.google/topics/google-cloud/protecting-our-g...
Earlier quoted context omitted.
It's very easy for a single admin with a single machine to provide 4 nines to a small group with a small load. But this usually scales exponentially.
Just because you can't do it doesn't make it impossible. A small team of 4-5 good ops people can scale a network of many thousand nodes, petabytes of storage, and terabits of network throughput. Tech stuff isn't hard. The biggest problem is the lack of capacity planning and project communication in tech today. Nimble startup is a euphemism for pure anarchy and chaos. No one wants to plan anything any more.
Earlier quoted context omitted.
https://www.kimsufi.com/us/en/ Granted, that's the bottom of the barrel (single disk, no IPKVM etc.), but $100 keeps you running for over a year. Better servers are easily available as well, usually a couple of times cheaper than AWS. Is this a US thing? Based on HN only, I'd never know there's anything between the public cloud and racks of own hardware that you have to wire up and maintain. I have a bunch of quad co…
HN has a lot of professionals. They can't run a business on a refurbished server without ECC and without RAID and without dual power supplies. Saying that they should run on kumsufi is like explaining to a wholesale company that they should use motorbikes instead of trucks, because motorbikes are cheaper.
We've been taught to build distributed system with unreliable componens and temporaral JIT eventual consistency.
Of course we can run production cloud-scale operations on unreliable systems. And single power supplies without ECC or RAID is pretty low on my list of things that cause outages. Most big hadoop/cassandra shops are running without raid and without redundant power.