Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: How do you update multiple disparate databases?

news.ycombinator.com

Post reply on HN
Print view View original

11–20 of 55 posts

Re: Ask HN: How do you update multiple disparate databases?

#11

Post by gwbas1c »

Honestly, the solution depends highly on your circumstances. Microsoft SQL has a concept of a distributed transaction, but I'm going to assume that "4 disparate legacy databases" aren't Microsoft SQL.

Here's the high-level approach I would take:

1: Understand which databases serve what, and what the needs for accuracy are. (For example, a database serving their public-facing website can probably be a few seconds behind compared to a database handling real money.)

2: Figure out which database is the source of truth. Update that database first.

3: Update the remaining three databases in a loop: Update the database, then check to make sure the update happened

You will probably need to add some state management where you put yourself into an "updating" state, so that if the update breaks in the middle, you can go back and recover.

----

Another approach is to refactor the applications that are critical (Those that handle money, not the public facing web site) to always read from the database you decide is the source of truth.

Re: Ask HN: How do you update multiple disparate databases?

#14

Post by chrisjc »

redis_mlc wrote: post #2

Banks use an ancient but powerful architecture called Source of Truth (SoT), or system of record. It's one of the first techniques developed to manage heterogeneous distributed databases. One master is picked as the final authority, then changes flow to other databases, replicas, etc. either using database tools or applications. Awareness of deadlines or how much latency (ie. when the updates are needed for each down…

> A practical example that solves a simple case is the recent release of a tool by Netflix.

Which tool is this?

Re: Ask HN: How do you update multiple disparate databases?

#15

Post by rorykoehler »

redis_mlc wrote: post #2

Banks use an ancient but powerful architecture called Source of Truth (SoT), or system of record. It's one of the first techniques developed to manage heterogeneous distributed databases. One master is picked as the final authority, then changes flow to other databases, replicas, etc. either using database tools or applications. Awareness of deadlines or how much latency (ie. when the updates are needed for each down…

I was in a client meeting a while back where one of their consultants corrected me with 'system of record' when I said source of truth. Your comment made me wonder where this distinction/difference was from? Do you know the history behind this?

Re: Ask HN: How do you update multiple disparate databases?

#17

Post by maps7 »

redis_mlc wrote: post #2

Banks use an ancient but powerful architecture called Source of Truth (SoT), or system of record. It's one of the first techniques developed to manage heterogeneous distributed databases. One master is picked as the final authority, then changes flow to other databases, replicas, etc. either using database tools or applications. Awareness of deadlines or how much latency (ie. when the updates are needed for each down…

What do non-banks use?

Re: Ask HN: How do you update multiple disparate databases?

#18

Post by in9 »

kjaftaedi wrote: post #3
redis_mlc wrote: post #2

Banks use an ancient but powerful architecture called Source of Truth (SoT), or system of record. It's one of the first techniques developed to manage heterogeneous distributed databases. One master is picked as the final authority, then changes flow to other databases, replicas, etc. either using database tools or applications. Awareness of deadlines or how much latency (ie. when the updates are needed for each down…

I get the feeling OP is the one being interviewed. Mortgage rates change frequently. The bank likely deals with this exact scenario on a regular basis and should have an established procedure in place.

You underestimate what how unknowledgeable a small financial institution can be.

Re: Ask HN: How do you update multiple disparate databases?

#19

Post by cabaalis »

lonelappde wrote: post #4

Send 4 API calls to update the 4 databases. Name the bank so we know not patronize patronize it.

Not sure why this would turn you off to an organization as this is a very common problem.

My money is on your suggestion being what most organizations actually do, though perhaps with a workflow process to manage it.

Legacy systems are hard.

Re: Ask HN: How do you update multiple disparate databases?

#20

Post by maps7 »

rorykoehler wrote: post #15
redis_mlc wrote: post #2

Banks use an ancient but powerful architecture called Source of Truth (SoT), or system of record. It's one of the first techniques developed to manage heterogeneous distributed databases. One master is picked as the final authority, then changes flow to other databases, replicas, etc. either using database tools or applications. Awareness of deadlines or how much latency (ie. when the updates are needed for each down…

I was in a client meeting a while back where one of their consultants corrected me with 'system of record' when I said source of truth. Your comment made me wonder where this distinction/difference was from? Do you know the history behind this?

It's probably just what they call it.
Post reply on HN