Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cell-based architecture for resilient payment systems

americanexpress.io

Post reply on HN
Print view View original

41–50 of 66 posts

Re: Cell-based architecture for resilient payment systems

#41

Post by orochimaaru »

otterley wrote: post #21
nightshift1 wrote: post #19

All i can see is a giant single point of failure called the Global Transaction Router.

GLBs aren’t SPOFs. They are typically deployed around the world redundantly, often using Anycast IPs or using DNS geographic and failover records, and are stateless. Think AWS Global Accelerator and Route 53 as an example. The architecture diagram is a high level simplification.

I don't think the global transaction router is a GLB. Having dabbled in this for high traffic telemetry gathering infrastructure, I will hazard a guess and say the "router" isn't a GLB.

The router needs to be shard-aware. It needs to know what data is where based on the request coming in so that it can route accurately. A GLB is DNS. It cannot be shard-aware because all it knows is the FQDN being resolved.

It can be a "router" if all the router needs to know is to resolve to the nearest data center or the nearest CDN. But at that point I have to ask the question - why does one need a cell-based architecture and can't it just be geo-redundant active-active failover across regions.

In any sense, the architecture itself isn't novel or new. It's documented here: https://docs.aws.amazon.com/wellarchitected/latest/reducing-.... It's the go to model if you're running a cloud.

Re: Cell-based architecture for resilient payment systems

#42

Post by cyberpunk »

tomjakubowski wrote: post #26
christophilus wrote: post #17

Earlier quoted context omitted.

“They reinvented Erlang OTP.” - Reddit

Don't know if Joe Armstrong ever said anything like it, but I would propose naming an Erlang/OTP analogue of Greenspun's tenth rule (the one about C projects containing ad-hoc, buggy implementations of Lisp) for him.

There is Virdings Rule:

Any sufficiently complicated concurrent program in another language contains an ad hoc informally-specified bug-ridden slow implementation of half of Erlang.

Re: Cell-based architecture for resilient payment systems

#43

Post by cyberpunk »

stevefan1999 wrote: post #14

Backing up would be hell

Backups in such a system are quite pointless; if losing 10 seconds of data means you lost 4000 transactions then periodic backups are invalid if not instantly than close to instantly.

The system I work on has such a property and the only real infra style approach is sync replication before responding to a caller and a delayed replica for delete/drop protections (say with a 2hr or more window).

Should also defend for this in your code (be able to reply from your initiation systems also etc)

Re: Cell-based architecture for resilient payment systems

#44

Post by genxy »

jijji wrote: post #40
Insimwytim wrote: post #25

Whole lot of nothing. This isn't about payment technologies, it's not about isolating transactions, it's about scaling the middle layer. What's worse it's not even explained what middle layer does. No info on how routing works, no info on data synchronization. Folks just learning Kubernetes and write extremely abstract stuff.

I agree, confusing and it seems like they are using the coined word "cell" to describe "container", but they really should say that instead of making up new words.

Making up new words is how you get to take ownership of existing concepts. This gets you promoted.

Re: Cell-based architecture for resilient payment systems

#45

Post by otterley »

orochimaaru wrote: post #41
otterley wrote: post #21

Earlier quoted context omitted.

GLBs aren’t SPOFs. They are typically deployed around the world redundantly, often using Anycast IPs or using DNS geographic and failover records, and are stateless. Think AWS Global Accelerator and Route 53 as an example. The architecture diagram is a high level simplification.

I don't think the global transaction router is a GLB. Having dabbled in this for high traffic telemetry gathering infrastructure, I will hazard a guess and say the "router" isn't a GLB. The router needs to be shard-aware. It needs to know what data is where based on the request coming in so that it can route accurately. A GLB is DNS. It cannot be shard-aware because all it knows is the FQDN being resolved. It can be…

Active/active without sharding is not a horizontal scaling model, and the blast radius of a fault is wide.

One can have GLBs that do routing. So long as the tenant-to-cell routing tables are consistent, it works fine. And those mappings tend not to change frequently.

Re: Cell-based architecture for resilient payment systems

#46

Post by otterley »

wbl wrote: post #28
otterley wrote: post #21

Earlier quoted context omitted.

GLBs aren’t SPOFs. They are typically deployed around the world redundantly, often using Anycast IPs or using DNS geographic and failover records, and are stateless. Think AWS Global Accelerator and Route 53 as an example. The architecture diagram is a high level simplification.

GLBs absolutely can be SPOF for certain kind of administrative mistakes.

If you’re counting administrative mistakes (human error), anything can fail. Let’s not shift the goalposts.

Re: Cell-based architecture for resilient payment systems

#47

Post by pyinstallwoes »

rdtsc wrote: post #39

Some of it sounds like it reinvented Erlang supervision trees https://learnyousomeerlang.com/supervisors . As a joke there we’re calling gen_severs “nanoservices”. Granted, that was mostly when microservices were the hot new thing.

Immediately my first thought as well. I still keep coming back to Erlang/Elixir/OTP as the best possible choice all things considered. I should use it more.

Re: Cell-based architecture for resilient payment systems

#49

Post by whazor »

physix wrote: post #9

Nobody uses Amex for payments, so the system isn't ever under high load. Just kidding! I find the idea quite good, and have to assume that the amount of payment fails they experience due to partitions/outages isn't very high and that the post-payment reconciliation and reclamation process gives them the liberty to rank availability a bit higher than correctness. One thing that looked a bit shaky was the interplay bet…

Amex is gaining popularity for acceptance

Re: Cell-based architecture for resilient payment systems

#50

Post by Scaled »

badlibrarian wrote: post #10

Ah yes, the financial services company that runs a travel agency, allows me to book my hotel and rental car weeks in advance, registers a hold for incidentals for both the hotel and car when I check in, then blocks the card when I try to buy dinner that night in that same hotel due to fraud detection. Last week it required me to take pictures of my face from multiple angles to regain membership privileges. I suspect…

I would have nope'd out so hard if they asked for face pictures.
Post reply on HN