181–190 of 190 posts
Earlier quoted context omitted.
Is there anything that can be done to mitigate that? E.g. give ssh network and the daemon top cpu priority?
Some equipment will auto-revert to a last known good configuration if you don't approve new changes within a window... though high CPU could lock that process up..
Earlier quoted context omitted.
> Debugging the problem was significantly hampered by failure of tools competing over use of the now-congested network. Man that's got to suck.
This is inconceivable ... they don't have an OOB management network?
Earlier quoted context omitted.
> I agree, but it’s pretty standard SLA verbiage (from the telco/bandwith provider days) to require the customer to request/register the SLA violation to benefit. FiOS has proactively given me per-day refunds of service without notification on my part. Weird to me that Verizon acts better than Google in this case.
Interesting, which kind of fios account? (residential/SMB/data center interconnect) That’s ideally how it should be!
I was curious to know how cascading failures in one region effected other regions. Impact was " ...increased latency, intermittent errors, and connectivity loss to instances in us-central1, us-east1, us-east4, us-west2, northamerica-northeast1, and southamerica-east1." Answer, and the root cause summarized: Maintenance started in a physical location, and then "... the automation software created a list of jobs to des…
At some point people realized servers are prone to failure. They then started deploying their system redundantly to multiple servers in the data center (AZ) to increase availability. This helped, but created consistency issues. To fix this people started building multi-server software systems, creating dependencies across servers that weakened overall availability. At some point people realized multi-server systems w…
Earlier quoted context omitted.
> I agree, but it’s pretty standard SLA verbiage (from the telco/bandwith provider days) to require the customer to request/register the SLA violation to benefit. FiOS has proactively given me per-day refunds of service without notification on my part. Weird to me that Verizon acts better than Google in this case.
It’s easier to determine “this line was down thus everyone along the line was also down” than what Google is facing.
Earlier quoted context omitted.
Is there anything that can be done to mitigate that? E.g. give ssh network and the daemon top cpu priority?
Yes - use taskset or isolcpus with other magic to put sshd on its own CPU core, or one core per CPU. Lots of HFT places do that.
Earlier quoted context omitted.
It’s easier to determine “this line was down thus everyone along the line was also down” than what Google is facing.
Google knows the affected regions and probably has very fine grained data around this. I mean, they can even tell you metrics about your instances, they don't have monitoring on their own infrastructure to determine impact radius?
And the part that even Google can’t know, even if they somehow can assemble all of the above: did it matter? Not all servers are created equal.
Small wonder they’re letting customers drive their own reimbursement process.
Earlier quoted context omitted.
Yes - use taskset or isolcpus with other magic to put sshd on its own CPU core, or one core per CPU. Lots of HFT places do that.
That doesn't help if the problem is a bandwidth congestion problem.
Earlier quoted context omitted.
SLACreditRequestsAAS? Who's with me, all I need is a co-founder and an eight million dollar series A round to last long enough that a cloud provider buys us up before they actually have to pay out a request!
This exists for Comcast and some other stuff, I'll ask my roommate what the service is called.