We have a distributed system where multiple services interact with each other, and we often run into inconsistent data states due to one or more services failing. What sort of methodologies can we use to identify such inconsistencies (e.g. anomaly detection systems)
Ask HN: Best way to monitor data incosistency in a distributed system
1–3 of 3 posts
Re: Ask HN: Best way to monitor data incosistency in a distributed system
#2In my experience you don't. Instead what you do is:
1. Make sure your services have a retry mechanism so even if it fails it keeps retrying.
2. Have the services write the data more than once. Have a job that runs them often.
Re: Ask HN: Best way to monitor data incosistency in a distributed system
#3Log IDs on the nodes then match them up when theres problems to work out the source, situation, etc...