Hardening the registers: A cascading failure of edge induced fault tolerance
1–10 of 80 posts
Re: Hardening the registers: A cascading failure of edge induced fault tolerance
#2No post body was provided.
Re: Hardening the registers: A cascading failure of edge induced fault tolerance
#3I presume "guest" means "customer"?
Re: Hardening the registers: A cascading failure of edge induced fault tolerance
#4Glad to see that a big US retailer like Target is using the same types of "de-facto" observability tools that I've been using for a while at all of my various employers over the last 5+ years - which are Grafana, Prometheus and Elastic Stack (specifically the Kibana UI for the logging analysis screenshot).
Re: Hardening the registers: A cascading failure of edge induced fault tolerance
#5I presume "guest" means "customer"?
Yes. Same way they call their employees "associates". I don't quite understand the rationale, but if I had to guess, "customer" and "employee" are a bit too on the nose, and they wish to cultivate a more human-feeling relationship between the customers, employees, and corporation in the minds of the former two groups.
Re: Hardening the registers: A cascading failure of edge induced fault tolerance
#6I presume "guest" means "customer"?
[deleted]
Re: Hardening the registers: A cascading failure of edge induced fault tolerance
#7I presume "guest" means "customer"?
Yes. Same way they call their employees "associates". I don't quite understand the rationale, but if I had to guess, "customer" and "employee" are a bit too on the nose, and they wish to cultivate a more human-feeling relationship between the customers, employees, and corporation in the minds of the former two groups.
Not just in the minds of the former two groups, but in the minds of their staff and leadership as well.
If you refer to your team members or employees as "associates" you're much more likely to treat them as equals.
Similarly, if you refer to your customers as "guests", you are much more likely to treat them as such rather than simply treating them as people in your store looking to spend money. It gets to the whole sense of trying to create an experience. As a store that sells a significant amount of home goods and goods for the home, referring to customers as guests instills the sense that employees are creating a home like experience for the customer.
Neurolinguistic programming isn't just for hippies. It's a very popular pseudoscience in corporate America.
Re: Hardening the registers: A cascading failure of edge induced fault tolerance
#8I presume "guest" means "customer"?
Some 90's thing that a couple retail stores started doing. Must have been popularized by whatever executives took advice from before Gary Vee and Seth Godin.
Re: Hardening the registers: A cascading failure of edge induced fault tolerance
#9I presume "guest" means "customer"?
Yes. Same way they call their employees "associates". I don't quite understand the rationale, but if I had to guess, "customer" and "employee" are a bit too on the nose, and they wish to cultivate a more human-feeling relationship between the customers, employees, and corporation in the minds of the former two groups.
Walmart has associates, Target has team members.
Re: Hardening the registers: A cascading failure of edge induced fault tolerance
#10This was really interesting both in exploring the architecture of a retail system and looking at how systems fail. Better to read about it and learn than to live it.
I'd call it a 4 hour outage because the initial "recovery" was a result of cashiers manually typing in prices for items. Then when load decreased and they discovered that scanning items worked again the problem came right back.
Maybe returning 404 for both a cache miss and a "there's no endpoint at this path" error is an issue too. For other status codes there's a distinction between temporary and permanent failure; e.g. 301 versus 302. It would've been good to use HTTP 400 Bad Request for the misconfigured URL and 404 for a cache miss.
In the 10% of stores with the early roll out of the config change the cache hit rate went to 0 right away, and that started 12 days before the outage. Alerts on cache hit rates and per-store alerts would've caught that.
Then there were 4 days where traffic to the main inventory micro-service in the data center jumped 3x which took it to what appears to be 80% of capacity. Load testing to know your capacity limits and alerts when you near that limit would've called out the danger.
Then during the outage when services slowed down due to too many requests they were taken out of rotation for failing health checks. Applying back pressure/load shedding could have kept those servers in active use so that the system could keep up.