Live data from Hacker News

Cell-based architecture for resilient payment systems

americanexpress.io

61–66 of 66 posts

Re: Cell-based architecture for resilient payment systems

#61
This matching up with sliced data, either sliced by recipient, or sliced by sender, so it makes sense to have primary state with cool backups. Particularly if the state is race-tolerant.

As long as event orderings are unimportant, or self resolvable this works really well.

e.g. if Events A,B arrive, but A+B => C and B+A => C, then as long as you durably record A, B, the end state is the same.

I'm not sure why "reroute" is a message instead of a response, I would expect it to be a failure response pruning the control flow. With the GTR being "default retry".

There's a lot of learned experience in that doc. Reading between the lines, both the logging and configuration systems have caused global outages (or near misses). Nifty to read.

This style of architecture fits with a "no global changes" and "never lose it all" approach to fault tolerance, accepting that there will be visits from "Mr. Cock-up" [1].

Very nice writeup.

[1] https://www.youtube.com/watch?v=D5r8xwu0l8w

Re: Cell-based architecture for resilient payment systems

#62
post #40

Whole lot of nothing. This isn't about payment technologies, it's not about isolating transactions, it's about scaling the middle layer. What's worse it's not even explained what middle layer does. No info on how routing works, no info on data synchronization. Folks just learning Kubernetes and write extremely abstract stuff.

I agree, confusing and it seems like they are using the coined word "cell" to describe "container", but they really should say that instead of making up new words.

if a "cell" is different from a "container", from the articles context, please explain how... This could be some special FPGA board too referred to again as a "cell", but they are extremely vauge about the definition.

Re: Cell-based architecture for resilient payment systems

#63
post #40

Whole lot of nothing. This isn't about payment technologies, it's not about isolating transactions, it's about scaling the middle layer. What's worse it's not even explained what middle layer does. No info on how routing works, no info on data synchronization. Folks just learning Kubernetes and write extremely abstract stuff.

I agree, confusing and it seems like they are using the coined word "cell" to describe "container", but they really should say that instead of making up new words.

If anything, it’s similar to a pod, not a container.

Re: Cell-based architecture for resilient payment systems

#64

As described, if one "cell" crashes, you re-route/retry everything on the other cell. Assuming your system is deterministic and the inputs stay the same, the second cell should now break as well. This reads to me like an attempt to patch a system that's already fucked beyond belief while pretending you're doing "engineering". Fancy implementation of a retry loop attempting to minimize downtime.

You know hardware failures can occur, right?

I worked as an SRE at a well-known monitoring company that used a similar architecture. It worked extremely well, and aside from one software SPOF (which still had a blast radius limited to that cell), we had very few large-scale production incidents compared to everywhere else I’ve worked at.

Even if there was a physical hardware failure (at the time, it ran on-prem, but it’s not like AWS is immune to this), every service modulo the aforementioned SPOF had redundancy, so we would have the datacenter techs replace that blade, which would provision itself and rejoin, zero downtime, just a temporary loss of redundancy. Even then, if we felt it necessary, we could shift customers into a different cell, though that did cause a brief outage for them, which would be coordinated ahead of time.

Re: Cell-based architecture for resilient payment systems

#66

As described, if one "cell" crashes, you re-route/retry everything on the other cell. Assuming your system is deterministic and the inputs stay the same, the second cell should now break as well. This reads to me like an attempt to patch a system that's already fucked beyond belief while pretending you're doing "engineering". Fancy implementation of a retry loop attempting to minimize downtime.

You know hardware failures can occur, right? I worked as an SRE at a well-known monitoring company that used a similar architecture. It worked extremely well, and aside from one software SPOF (which still had a blast radius limited to that cell), we had very few large-scale production incidents compared to everywhere else I’ve worked at. Even if there was a physical hardware failure (at the time, it ran on-prem, but…

[deleted]
Post reply on HN