Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Visualizing Meltdown on AWS

blog.appoptics.com

Post reply on HN
Print view View original

61–70 of 72 posts

Re: Visualizing Meltdown on AWS

#61

Post by scurvy »

user5994461 wrote: post #49
scurvy wrote: post #38

Earlier quoted context omitted.

Just because you can't do it doesn't make it impossible. A small team of 4-5 good ops people can scale a network of many thousand nodes, petabytes of storage, and terabits of network throughput. Tech stuff isn't hard. The biggest problem is the lack of capacity planning and project communication in tech today. Nimble startup is a euphemism for pure anarchy and chaos. No one wants to plan anything any more.

The same infra could be managed by a single guy if it were in the cloud.

No, because that "guy" would want a vacation and would suddenly become the most important person in the company. You're running teams of people, not cogs. Once you realize that, you'll also realize that cloud services are like 5-6x the total cost.

Cloud services are not efficient or cheap, they're convenient for the unprepared and haphazard management style that VC's love.

Re: Visualizing Meltdown on AWS

#62

Post by plandis »

jacksmith21006 wrote: post #42
pathseeker wrote: post #17

Earlier quoted context omitted.

Google likely only wrote a blog post about it because they were able to find a way to brag that there was effectively no performance hit. They have never written any blog posts IIRC explaining bad/unpredictable performance on GCE

Do they not deserve to brag about it? Heck they among others are who found the flaws as well as Broadpwn, Cloudbleed, Heartbleed among others and deserve credit for the work they have done, imo. Think the constant ragging on Google that appears on HN is becoming a little too much.

It was Google who discovered the vulnerabilities in the first place. How long did they internally wait to give themselves and advantage before disclosing these? I'm honestly asking I don't know.

Re: Visualizing Meltdown on AWS

#63

Post by 013a »

user5994461 wrote: post #46
013a wrote: post #44

Earlier quoted context omitted.

Technically, you pay for both time (hours on) and CPU Usage (instance tier). Its not like different instance tiers (at least in the same class) use fundamentally more or less powerful processors. They all use the same processors, you just get more or less of it depending on what you pay. Conceptually it is "pay as you use" by CPU usage, but just rounded into buckets by instance tier. Of course, there's a lot of under…

>>> Its not like different instance tiers (at least in the same class) use fundamentally more or less powerful processors. They all use the same processors, you just get more or less of it depending on what you pay. There is a variety of CPUs. You can "cat /proc/cpuinfo" to see what you got. >>> most applications can't switch instance tiers easily to adapt to demand (though some certainly can). Most applications can…

> There is a variety of CPUs.

There certainly are, but generally they are the same within the same "class" (ie t2, c5, etc). Example:

- C5: 3.0Ghz custom Xeon processors. - C4: Xeon E5-2666 v3 - C3: Xeon E5-2680 v2 - X1/X1e: Xeon E7-8880 v3 - R4: Xeon E5-2686 v4 - R3: Intel Xeon E5-2670 v2

Etc. The only exception I'm aware of are the burstable instances; AWS's documentation doesn't explicitly guarantee a CPU bin there, it just says "high frequency Xeon".

The difference between an m5.large and an m5.xlarge isn't that you're getting a "faster" processor (meaning higher gigahertz or newer architecture). You're just getting "more" of the same processor (more cores). This is different than on, say, GCP, where you just ask for cores and you can specify if you specifically want a broader generation of Xeon (like Broadwell), but you can't be guaranteed specific chips.

My intention behind saying that most applications can't switch instance tiers is more to point out that most applications aren't prepared to handle node failure, not that there is something intrinsic to the node types which stops them from being able to switch (that'd be much more rare).

Re: Visualizing Meltdown on AWS

#64

Post by rdtsc »

boulos wrote: post #48
rdtsc wrote: post #13

That's one interesting aspect of these issues and mitigations is that performance really depends on the workload. Just because Google saw little performance impact on their servers, doesn't mean your application won't see. Or because someone said their CPU usage went up 2x doesn't mean it will go up for you. On an unrelated note, kind of wish Meltdown had been discovered and exposed separately from Spectre. Intel has…

Disclosure: I work on Google Cloud. Just a nit, we said [1] that not only are our own applications in production doing fine (even against Variant 2) but also we haven’t been inundated with support calls over the last few months while mitigations were silently rolled out at the host kernel and hypervisor layer. So this class of “Hey, my instances are suddenly way slower, I didn’t do anything” isn’t happening on GCE. T…

> but also we haven’t been inundated with support calls over the last few months

Sorry, I didn't mean to imply that it was Google's customers specifically who should measure and be suspicious. I meant in general, say someone running a service on bare metal outside of any public cloud for example drawing this conclusion - "Google measured the impact of these bugs and mitigations and they didn't see any a significant performance regressions so I probably don't have to worry either".

Also (since you disclosed you work on GCP), I like how Google sponsored Project Zero. Fantastic work and I am sure it will be a great return on that investment. I can certainly see someone thinking about that when deciding to go with GCP vs other solution.

Re: Visualizing Meltdown on AWS

#65

Post by forgot-my-pw »

patrickxb wrote: post #2

It would be nice if AWS could write something official about what they are doing. I've been noticing major performance changes in our instances and have no idea if it is related to Meltdown or something else. Google released a blog post specifically on performance: https://blog.google/topics/google-cloud/protecting-our-googl... It would be nice to have similar transparency from AWS.

Amazon will probably earn more from autoscaling due to the slowdown.

Re: Visualizing Meltdown on AWS

#66

Post by _msw_ »

taf2 wrote: post #41
mike_heffner wrote: post #40

Would love to know if anyone else had data on: * Impact on M5/C5 instances over similar time period, any difference with the Nitro hypervisor? * Were Dedicated instances ( https://aws.amazon.com/ec2/purchasing-options/dedicated-inst... ) patched as well? * Other examples of software that adapted batching performance automatically with increase in call latency.

We had a lot of m5 and c5 servers randomly die. It was as if someone was running chaos monkey from Netflix in our VPCs...

Could you send a list of instance IDs and timeframes where you saw this?

Re: Visualizing Meltdown on AWS

#67

Post by amazingman »

lykr0n wrote: post #31
amazingman wrote: post #19

Earlier quoted context omitted.

You forgot to type 5 or 6 zeros there.

If you know what you are doing, it is leaps and bounds cheaper to run your own hardware (co-located, rented from soneone else). The only issue is latency on scaling out (hours), but if you are halfway decent with trend lines you can preempt this.

And what if the delta between your upper and lower daily “trend lines” is measured in millions of requests per hour? Per second? We can leave off weekly/seasonal trends for now, and keep it nice and easy for you.

The utter lack of imagination that I see on HN when people are judging others’ technical decisions is kind of hilarious.

Re: Visualizing Meltdown on AWS

#68

Post by amazingman »

scurvy wrote: post #61
user5994461 wrote: post #49

Earlier quoted context omitted.

The same infra could be managed by a single guy if it were in the cloud.

No, because that "guy" would want a vacation and would suddenly become the most important person in the company. You're running teams of people, not cogs. Once you realize that, you'll also realize that cloud services are like 5-6x the total cost. Cloud services are not efficient or cheap, they're convenient for the unprepared and haphazard management style that VC's love.

You should review the difference between CapEx and OpEx, and specifically focus on why companies might prefer to pay “5-6x the total cost” when that cost comes in the form of OpEx.

You may still disagree in the end, but you should come to the conclusion that it is not categorically stupid for some companies to choose to pay much more over time so that they can pay a lot less right now.

Re: Visualizing Meltdown on AWS

#69

Post by amazingman »

gnosek wrote: post #24
amazingman wrote: post #19

Earlier quoted context omitted.

You forgot to type 5 or 6 zeros there.

https://www.kimsufi.com/us/en/ Granted, that's the bottom of the barrel (single disk, no IPKVM etc.), but $100 keeps you running for over a year. Better servers are easily available as well, usually a couple of times cheaper than AWS. Is this a US thing? Based on HN only, I'd never know there's anything between the public cloud and racks of own hardware that you have to wire up and maintain. I have a bunch of quad co…

Yeah, and AWS has a free tier. $0/month is better than $100/month, right?

Re: Visualizing Meltdown on AWS

#70

Post by stephengillie »

user5994461 wrote: post #49
scurvy wrote: post #38

Earlier quoted context omitted.

Just because you can't do it doesn't make it impossible. A small team of 4-5 good ops people can scale a network of many thousand nodes, petabytes of storage, and terabits of network throughput. Tech stuff isn't hard. The biggest problem is the lack of capacity planning and project communication in tech today. Nimble startup is a euphemism for pure anarchy and chaos. No one wants to plan anything any more.

The same infra could be managed by a single guy if it were in the cloud.

Only by abstracting away hundreds of teams that manage the actual hardware.
Post reply on HN