201–209 of 209 posts
I'm curious about big outages like this in big internet corps. Does anyone know if SREs in Europe fixed the problem or it relied on people in Mountain View? When an outage this big hits do devs get involved? I've worked as an SRE and it sucks to be fixing developer's bugs in the middle of the night.
Earlier quoted context omitted.
> Don’t assume that is the case. What I mean is, it's going to have limits of some sort, right? The world is finite...
Yes, everything has limits. Where Google says "quota system", for normal people that means "buy another computer"; you have hit your quota when you're out of memory / cpu cycles / disk. At Google, they have some extra computers sitting around, but it's still not infinite. Quota is a way of hitting some sort of limit before every atom in the Universe becomes a computer on which to run your program. I don't think there…
There's a difference here; helmets are personal safety equipment, which is the proper approach: monitor and manage yourself, don't rely on external barriers. But did-you-know that a statistically significant proportion of drivers change their behaviour around riders wearing helmets? [1] (That's not a reason to not wear helmets, everyone should ATGATT; it's a reason to change driver behaviour through other incentives).
We cannot deny the existence of moral hazards. If you want to nullify a well-understand, thoroughly documented, and strongly correlated statistical behaviour, something has to replace it. Google would, apparently, prefer to cover a hard barrier with soft padding. That might help ... until the padding catches fire.
To your example, writing to disk until the OS reports "there are no more sectors to allocate" just means no-one was monitoring the disk consumption, which would be embarrassing, since that is systems administration 101. Or projecting demand rate for more storage, which is covered in 201, plus an elective of haggling with vendors, and teaching developers about log rotation, sharding, and tiered archive storage.
Actionable monitoring and active management of infrastructure beats automatic limits, every time, and I've always seen it as a sign of organisational maturity. It's the corporate equivalent of taking personal responsibility for your own safety.
[1] http://www.drianwalker.com/overtaking/overtakingprobrief.pdf
I've been bitten by quota grace periods before, they're the "buffer bloat" of cloud platform management systems. You build something, it seems fine, and then a month later it keels over. My tiny disaster was caused by Amazon EFS. They provide a performance quota grace period of a month, during which all tests passed with flying colours. Turns out that EFS rations out IOPS per GB of stored data, and this particular ap…
Even worse are undocumented quotas. Probably not quite the same thing, but I've had such an issue with a SaaS API for extracting data for a cloud system. It has an unpublished limit on queries. Writing a library of code to use it, I was fine. All tests passed. Running a full day's data pull & integration had sporadic failure: It would run w/o issue for half the job, then have seeming random errors on some requests. I…
They used to advertise 100 per second, but then they pushed a big update that doubled the read QPS limit, and conveniently left out the reduction in LIST qps from the docs.
Earlier quoted context omitted.
Ooh boy. Undocumented limitations caused major problems with us with AWS SQS. It is now in the documentation (maybe because I complained to our AWS rep), but SQS has pool of 20k messages that it will pull from when serving requests. If you are using a FIFO queue and you have 20k messages with the same message group ID in that buffer, then you are unable to process any other message regardless of how large the queue i…
Pretty scary that it just breaks without any error messages, and without an easy way to clean up the damage. Queue overflows are not uncommon. "For FIFO queues, there can be a maximum of 20,000 inflight messages (received from a queue by a consumer, but not yet deleted from the queue). If you reach this quota, Amazon SQS returns no error messages. If your queue has a large backlog of 20,000 or more messages with the…
Earlier quoted context omitted.
Ooh boy. Undocumented limitations caused major problems with us with AWS SQS. It is now in the documentation (maybe because I complained to our AWS rep), but SQS has pool of 20k messages that it will pull from when serving requests. If you are using a FIFO queue and you have 20k messages with the same message group ID in that buffer, then you are unable to process any other message regardless of how large the queue i…
Wow, that's a scary story.
I'm curious about big outages like this in big internet corps. Does anyone know if SREs in Europe fixed the problem or it relied on people in Mountain View? When an outage this big hits do devs get involved? I've worked as an SRE and it sucks to be fixing developer's bugs in the middle of the night.
I assume this is in the SRE book, but a tier one product like the identity service will have global SRE coverage (i.e. at least three SRE teams so that there is always an SRE group for whom it is daytime holding the pager). Devs are often involved in diagnosis, but are less often required for mitigation, as the mitigation is almost always to revert whatever change caused the problem. This is a simplification of cours…
To be honest, I never understood the point of companies publishing post mortems after outages. What is it supposed to accomplish? We the users don’t understand their infrastructure anyway, how are we expected to understand the post mortem? Besides, as the users why should we care why things went down? The fact is, they did. And that’s the only thing that matters. I don’t feel better knowing WHY I lost my services. I…
For example Interactive Brokers was down completely for almost the whole trading session and they just said that they are sorry and that it was their database vendor who messed up. No more details, nothing. Just "we take the quality and resiliency of our systems very seriously".
Earlier quoted context omitted.
It doesn’t have to be this way, but that’s partly a matter of culture. By aspiring to present/think/act as a monoplatform, Google risks substantially increasing the blast radius of individual component failure. A global quota system mediating every other service sounds both totally on brand, and also the antithesis of everything I learned about public cloud scaling at AWS. There we made jokes, that weren’t jokes, abo…
I often hear 'aim for elimination of global dependencies', but the reality is that there is no way around global dependencies. AWS STS or IAM is just as global as google's. The difference is that google more often builds with some form of guaranteed read-after-write consistency, while AWS is more often 'fail open'. For example, if you remove a permission from a user in GCP, you are guaranteed consistency within 7 min…