Re: transition to micro services (from monolith). I’m surprised a network so sensitive to latency (as are payment networks), was able to achieve their latency SLAs with micro services. Maybe Amex being a closed-loop network helps with latency?
Since when were payment networks latency sensitive? It’s usually 2 or more seconds to even get a payment up on the card terminal from the merchant POST system, then 2-5 seconds more from card presentation to getting approval back.
Migrating the American express payment network, twice
31–40 of 41 posts
Re: Migrating the American express payment network, twice
#32This is cool, but a part of me has to be flip, and humanity is kind of hilarious to me. All of this amazing infrastructure and hard work to essentially just very carefully and accurately subtract one number from another in a way that makes auditing easy.
"Rebuilding the full payments network from scratch was a significant, multi-year effort. It involves complex processing logic, extensive edge cases, and exception handling." Quite apart from fraud/abuse prevention, I expect part of this is stuff like juggling all the different rewards and points systems plus handling time-based offers ("spend $x at y retailer by z date, get $10 cash back" type thing) plus ensuring th…
?!? Are you saying that Amex is the first Canadian credit card you've used to let you make the payment from on its app, with Amex pulling the funds from your bank? Other Canadian credit cards all require you to "push" payments to them via your bank's billpay? This astounds me.
Re: Migrating the American express payment network, twice
#33but also
> Latency, throughput, and resiliency characteristics had to remain consistent, and in some cases improve.
> Payment requests could not be dropped, delayed, or left unanswered.
what else would an "anomaly" be?
Re: Migrating the American express payment network, twice
#34migrating services is easier if your DB supports those things and is more oriented around AP
now a lot of people just jump straight to microservices without considering what their DB can do for them, or if at times they need a relational DB, or if their use case can be served by a hybrid Db.
if you see most of the talks at Monster Scale [0]: They're around DBs
Re: Migrating the American express payment network, twice
#35Re: Migrating the American express payment network, twice
#36Earlier quoted context omitted.
"Rebuilding the full payments network from scratch was a significant, multi-year effort. It involves complex processing logic, extensive edge cases, and exception handling." Quite apart from fraud/abuse prevention, I expect part of this is stuff like juggling all the different rewards and points systems plus handling time-based offers ("spend $x at y retailer by z date, get $10 cash back" type thing) plus ensuring th…
>- an in-app button to debit my bank for the balance without me having to go to my bank's app to send a bill pay. ?!? Are you saying that Amex is the first Canadian credit card you've used to let you make the payment from on its app, with Amex pulling the funds from your bank? Other Canadian credit cards all require you to "push" payments to them via your bank's billpay? This astounds me.
Same with everything else: power, water, taxes, mortgage, all of these I have to look up the number and arrange a bill-pay push, I think only my mobile and internet provider are able to just debit my bank acct directly.
Re: Migrating the American express payment network, twice
#37Earlier quoted context omitted.
It's not simple though. In that 140ms the network is checking fraud rules, validating the card, checking available credit, applying rewards logic, and routing across multiple parties. The actual subtract-one-number-from-another takes microseconds. The rest is trust verification across organizational boundaries — which is the hard part of any payment system.
At best it’s checking available credit. All the other stuff is done after the fact. The idea that any banking transaction involves “subtracting one number from another” is so wrong it’s barely worth engaging with.
Re: Migrating the American express payment network, twice
#38Cmd+F "Kubernetes". Oh Jesus Christ.
Re: Migrating the American express payment network, twice
#39Earlier quoted context omitted.
I’m fairly certain that’s a POS issue, not the payment network. I’ve heard anecdotally that it’s Anyone, please correct me if you know better.
It's definitely not 140ms round-trip. Issuer processors typically have a ~2.5s window for approval. In practice, the POS sends a message to the acquirer processor -> hits the network -> is sent to the issuer processor, and back again. https://medium.com/wharton-fintech/the-anatomy-of-the-swipe-...
In EU they use of offline PIN was used massively before PSD2 and contactless, that made the terminal request during the time it took for validating the transaction online, and basically as soon as the PIN was ok'ed by the card that confirmed the transaction. That gave a perception of speed.
Now it's basically online PIN mostly or contactless, but that means you perceive a "wait for an ok", that you had before but was masked by the PIN capture and check on device/card.
So we went a bit backwards for cards, but wallets like ApplePay went a bit forward. You win some you lose some I guess
Re: Migrating the American express payment network, twice
#40Re: transition to micro services (from monolith). I’m surprised a network so sensitive to latency (as are payment networks), was able to achieve their latency SLAs with micro services. Maybe Amex being a closed-loop network helps with latency?
There's ton of logic on non-critical path that can be extracted to other microservices and called asynchronously - settlements, refunds, rewards, all management and reporting functionalities - to name just a few.