91–100 of 114 posts
Earlier quoted context omitted.
Right, but they didn't recover speedily. To have the cluster in such a state for so long sounds like poor monitoring to me because this can knowingly interfere with an election later.
The health check said it was ok. How would they know it needed to be recovered? The fault was the bad health check. Not the process.
Earlier quoted context omitted.
How do you have a ATM thats not networked?
Same user (sorry I guess I didn't enter my password carefully as I can't log in.) Well I mean they're not exactly on the Internet with an IP address and no firewall, are they? (Or they would have been compromised already.) Whatever it is, it must be separated off as an "insecure enclave". So that's why I'm wondering about this technique. You don't just miss out on security updates, you miss performance and architectu…
Earlier quoted context omitted.
Both have had plenty of downtime: https://www.ft.com/content/1fd2a066-860f-11e8-a29d-73e3d4545... I suspect they sometimes 'fail open' (ie. allow all payments through and reconcile later) too.
No they don't. If I sell a diamond ring for $20k and Visa passes that the card is valid but it's not. The buyer just got a free $20k ring. Card could be expired, cancelled, or not have enough balance. The merchant must be paid, their processor has to pay them, the bank that issued the card must provide that credit until the card holder pays it back. If the card was expired or a card with a $10 balance. The card holde…
Why don't they call 'significantly elevated error rates' an 'outage' instead?
Because "A substantial majority of API requests during these windows failed. " implying that there was not a complete outage. I don't understand why people demand the usage of incorrect language.
The fact that you needed to qualify “outage” with “complete” clearly means the word on its own is not incorrect for cases where a system was “only” mostly unavailable rather than completely so.
> I don't understand why people demand the usage of incorrect language.
The irony.
Earlier quoted context omitted.
Same user (sorry I guess I didn't enter my password carefully as I can't log in.) Well I mean they're not exactly on the Internet with an IP address and no firewall, are they? (Or they would have been compromised already.) Whatever it is, it must be separated off as an "insecure enclave". So that's why I'm wondering about this technique. You don't just miss out on security updates, you miss performance and architectu…
Secure .. enclave? I'm sorry but I think you're throwing buzzwords around hoping to hit a homerun here.
So the legit question is, can insecure systems (e.g. ancient mainframes) be wrapped by a security layer (WAF, etc.) to get better uptime than patching an exposed system?
Earlier quoted context omitted.
(Stripe infra lead here) This was a focus in our after-action review. The nodes responded as healthy to active checks, while silently dropping updates on their replication lag, together this created the impression of a healthy node. The missing bit was verifying the absence of lag updates. (Which we have now.)
I am a curious and very amateur person, but do you think that if "100%" uptime were your goal, this: "[Three months prior to the incident] We upgraded our databases to a new minor version that introduced a subtle, undetected fault in the database’s failover system." could have been prevented if you had stopped upgrading minor versions, i.e. froze on one specific version and not even applied security fixes, instead re…
But customers want new features, so Stripe does changes.
Why don't they call 'significantly elevated error rates' an 'outage' instead?
(Stripe CTO here) That's a reasonable question. We wrote this RCA to help our users understand what had happened and to help inform their own response efforts. Because a large absolute number of requests with stateful consequences (including e.g. moving money IRL) succeeded during the event, we wanted to avoid customers believing that retrying all requests would be necessarily safe. For example, users (if they don’t…
Earlier quoted context omitted.
Secure .. enclave? I'm sorry but I think you're throwing buzzwords around hoping to hit a homerun here.
No, it's a fair question. The word "enclave" has a general meaning in English as a state surrounded entirely by another, or metaphorically a zone with some degree of isolation from its surroundings. So the legit question is, can insecure systems (e.g. ancient mainframes) be wrapped by a security layer (WAF, etc.) to get better uptime than patching an exposed system?
Earlier quoted context omitted.
Mainframes. > Visa, for example, uses the mainframe to process billions of credit and debit card payments every year. > According to some estimates, up to $3 trillion in daily commerce flows through mainframes. https://www.share.org/blog/mainframe-matters-how-mainframes-... https://blog.syncsort.com/2018/06/mainframe/9-mainframe-stat... https://www.ibm.com/it-infrastructure/servers/mainframes
Specifically they run IBM zTPF on their mainframes, which is also used by airlines. Some installations have uptimes measured in decades. https://www.ibm.com/it-infrastructure/z/transaction-processi...
Earlier quoted context omitted.
Specifically they run IBM zTPF on their mainframes, which is also used by airlines. Some installations have uptimes measured in decades. https://www.ibm.com/it-infrastructure/z/transaction-processi...
It's rarely the hardware that fails, it's more often due to software. So I wonder what the software that's running on mainframes does differently than the software that's written for ordinary computers.