Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Hardening the registers: A cascading failure of edge induced fault tolerance

tech.target.com

Post reply on HN
Print view View original

41–50 of 80 posts

Re: Hardening the registers: A cascading failure of edge induced fault tolerance

#41

Post by joeframbach »

yardstick wrote: post #13

> Although Autobahn contained all the item data, the 404 responses were interpreted by the SDM Proxy as an indicator that the item was missing in Autobahn and the SDM Proxy retried the request to the central ILS API in the data centers. This is why I never design web APIs to use the HTTP status code to indicate the application response. Always embed the application response within the HTTP payload. It should be indep…

The proper response should have been 400 Bad Request, not 404 Not Found. Because the client's request contained bad request data. Not that the item wasn't found.

Re: Hardening the registers: A cascading failure of edge induced fault tolerance

#42

Post by drjasonharrison »

at_a_remove wrote: post #33
tylerrobinson wrote: post #22

Earlier quoted context omitted.

Can you give an example of what you mean?

So when I ran the university website, the homepage naturally had links to other sites. One guy had this inflated sense of importance. If there weren't a lot of clicks over to his site, we should MAKE THE LINK BIGGER because people weren't seeing it. If clicks to his site went up, we should MAKE THE LINK BIGGER because it is that important. His flowchart had only one distinct outcome: MAKE THE LINK BIGGER. All of the…

Thank you.

Re: Hardening the registers: A cascading failure of edge induced fault tolerance

#43

Post by kerbs »

yardstick wrote: post #13

> Although Autobahn contained all the item data, the 404 responses were interpreted by the SDM Proxy as an indicator that the item was missing in Autobahn and the SDM Proxy retried the request to the central ILS API in the data centers. This is why I never design web APIs to use the HTTP status code to indicate the application response. Always embed the application response within the HTTP payload. It should be indep…

Targets mobile apps were down over Black Friday many years back for a very similar reason – logic done on status codes.

A 403 in the API had a very specific meaning, and when the proxy layer started returning 403s everyone had a really bad time.

(That was a long day)

Re: Hardening the registers: A cascading failure of edge induced fault tolerance

#44

Post by kevan »

jaywalk wrote: post #37
bombcar wrote: post #26

Earlier quoted context omitted.

The total size of all item data for a store like Target can't be much more than what, a few gigabytes? Or at least the "UPC -> Price" dataset. So download the whole dataset each night, if you can't get delta changes to work. And if that had failed somehow, it would have been noticed immediately upon the new code roll-out. The internet was designed to be extremely resilient to host/route losses, we've made it so relia…

Their system was basically designed the way you're saying, with a fallback to grab the data from the central location if it's missing locally. What you're asking for is the same system without a fallback, which doesn't make any sense.

It's counterintuitive but when you're dealing with distributed systems lots of things are: https://aws.amazon.com/builders-library/avoiding-fallback-in...

Re: Hardening the registers: A cascading failure of edge induced fault tolerance

#45

Post by a-dub »

yardstick wrote: post #13

> Although Autobahn contained all the item data, the 404 responses were interpreted by the SDM Proxy as an indicator that the item was missing in Autobahn and the SDM Proxy retried the request to the central ILS API in the data centers. This is why I never design web APIs to use the HTTP status code to indicate the application response. Always embed the application response within the HTTP payload. It should be indep…

also maybe counting any sort of retry or fallback as a trackable and alertable failure.

auto/silent fallbacks seem like clever ways of avoiding beeps and remaining resilient against failures in supporting systems, but in practice it they always tend to just cover up real issues until it's too late.

i think the ideal is to have a nice easily included retry library that includes reporting/alerting, configurable backoff schemes and logging that can be used everywhere on things that can have transient failures.

Re: Hardening the registers: A cascading failure of edge induced fault tolerance

#46

Post by marcosdumay »

kerbs wrote: post #43
yardstick wrote: post #13

> Although Autobahn contained all the item data, the 404 responses were interpreted by the SDM Proxy as an indicator that the item was missing in Autobahn and the SDM Proxy retried the request to the central ILS API in the data centers. This is why I never design web APIs to use the HTTP status code to indicate the application response. Always embed the application response within the HTTP payload. It should be indep…

Targets mobile apps were down over Black Friday many years back for a very similar reason – logic done on status codes. A 403 in the API had a very specific meaning, and when the proxy layer started returning 403s everyone had a really bad time. (That was a long day)

> A 403 in the API had a very specific meaning

And that meaning wasn't "you are authenticated as a user that can not access this resource"?

Re: Hardening the registers: A cascading failure of edge induced fault tolerance

#47

Post by bombcar »

kevan wrote: post #44
jaywalk wrote: post #37

Earlier quoted context omitted.

Their system was basically designed the way you're saying, with a fallback to grab the data from the central location if it's missing locally. What you're asking for is the same system without a fallback, which doesn't make any sense.

It's counterintuitive but when you're dealing with distributed systems lots of things are: https://aws.amazon.com/builders-library/avoiding-fallback-in...

Exactly - they had a fallback system that worked well enough for the testing to pass, but not well enough for the main system to operate on it.

Interestingly enough the Amazon example there is basically exactly what happened to Target.

Re: Hardening the registers: A cascading failure of edge induced fault tolerance

#48

Post by throwaway894345 »

tornato7 wrote: post #24
yardstick wrote: post #13

> Although Autobahn contained all the item data, the 404 responses were interpreted by the SDM Proxy as an indicator that the item was missing in Autobahn and the SDM Proxy retried the request to the central ILS API in the data centers. This is why I never design web APIs to use the HTTP status code to indicate the application response. Always embed the application response within the HTTP payload. It should be indep…

Reminds me of an online ordering app at my university - when the API went down one day the app started reporting to everyone that the wait time for their food was “503 minutes”

That's just awful programming, and the parent's suggestion wouldn't have addressed that. It would have instead resulted in "1000 minutes" or whatever their status code was.

Re: Hardening the registers: A cascading failure of edge induced fault tolerance

#49

Post by jkaptur »

It's very interesting that by building a system that's more resilient and reliable:

> high profile processes (such as POS) implement their own fallback processes to handle the possibility of issues with the SDM system in store. In the case of item data, the POS software on each register is capable of bypassing the SDM Proxy and retrying its request directly to the ILS API in the data centers.

... the system as a whole became much more complex and difficult to observe. The system was running in a degraded, abnormal, less-tested, fallback mode for days without anyone caring.

This is also a point about the normalization of deviance. When there is a background rate of the POS using the fallback path, who is to say how important an increase in that rate might be?

Re: Hardening the registers: A cascading failure of edge induced fault tolerance

#50

Post by EvanAnderson »

InCityDreams wrote: post #3

I presume "guest" means "customer"?

My wife worked for Darden Restarurants for awhile and corporate training materials always referred to customers as "guests", too.

On one level I suppose it's just silly terminology, but it grates with me. I guess it's supposed to imply some kind of familiar relationship, free of the gauche trappings of economics. To me a customer demands more attention than a "guest".

It shocks me how many people don't recognize that their employer wouldn't exist if not for customers. That should be front-and-center in the minds of anyone working for a for-profit entity. I don't think there's anything gauche about economics.

Post reply on HN