Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Hardening the registers: A cascading failure of edge induced fault tolerance

tech.target.com

Post reply on HN
Print view View original

21–30 of 80 posts

Re: Hardening the registers: A cascading failure of edge induced fault tolerance

#21

Post by bombcar »

haroldl wrote: post #10

This was really interesting both in exploring the architecture of a retail system and looking at how systems fail. Better to read about it and learn than to live it. I'd call it a 4 hour outage because the initial "recovery" was a result of cashiers manually typing in prices for items. Then when load decreased and they discovered that scanning items worked again the problem came right back. Maybe returning 404 for bo…

It seems absolutely insane to me that a system was designed and developed that allows taking down all registers in the country at once. I would have thought it would be designed to be much more "batch" oriented and the worst that could happen is you lose price updates and sales info unto the batches can get through again.

Re: Hardening the registers: A cascading failure of edge induced fault tolerance

#22

Post by tylerrobinson »

at_a_remove wrote: post #12

I don't work at that level, or even want to, but I did detect a dark pattern that I often complain about, but have never managed to get people to pay attention to: do not collect data unless you have attached to it a decision with two or more distinct outcomes based on that data.

Can you give an example of what you mean?

Re: Hardening the registers: A cascading failure of edge induced fault tolerance

#23

Post by jaywalk »

yardstick wrote: post #13

> Although Autobahn contained all the item data, the 404 responses were interpreted by the SDM Proxy as an indicator that the item was missing in Autobahn and the SDM Proxy retried the request to the central ILS API in the data centers. This is why I never design web APIs to use the HTTP status code to indicate the application response. Always embed the application response within the HTTP payload. It should be indep…

I think a better solution would be to use the 404 code but include a body with the detailed error. That way the response to an invalid URL looks different from an actual item not being found.

Re: Hardening the registers: A cascading failure of edge induced fault tolerance

#24

Post by tornato7 »

yardstick wrote: post #13

> Although Autobahn contained all the item data, the 404 responses were interpreted by the SDM Proxy as an indicator that the item was missing in Autobahn and the SDM Proxy retried the request to the central ILS API in the data centers. This is why I never design web APIs to use the HTTP status code to indicate the application response. Always embed the application response within the HTTP payload. It should be indep…

Reminds me of an online ordering app at my university - when the API went down one day the app started reporting to everyone that the wait time for their food was “503 minutes”

Re: Hardening the registers: A cascading failure of edge induced fault tolerance

#25

Post by jaywalk »

bombcar wrote: post #21
haroldl wrote: post #10

This was really interesting both in exploring the architecture of a retail system and looking at how systems fail. Better to read about it and learn than to live it. I'd call it a 4 hour outage because the initial "recovery" was a result of cashiers manually typing in prices for items. Then when load decreased and they discovered that scanning items worked again the problem came right back. Maybe returning 404 for bo…

It seems absolutely insane to me that a system was designed and developed that allows taking down all registers in the country at once . I would have thought it would be designed to be much more "batch" oriented and the worst that could happen is you lose price updates and sales info unto the batches can get through again.

If the local system doesn't have the item data (or thinks it doesn't have the item data, because it's looking in the wrong place) where exactly is it supposed to get the item data from if not the central system?

Re: Hardening the registers: A cascading failure of edge induced fault tolerance

#26

Post by bombcar »

jaywalk wrote: post #25
bombcar wrote: post #21

Earlier quoted context omitted.

It seems absolutely insane to me that a system was designed and developed that allows taking down all registers in the country at once . I would have thought it would be designed to be much more "batch" oriented and the worst that could happen is you lose price updates and sales info unto the batches can get through again.

If the local system doesn't have the item data (or thinks it doesn't have the item data, because it's looking in the wrong place) where exactly is it supposed to get the item data from if not the central system?

The total size of all item data for a store like Target can't be much more than what, a few gigabytes? Or at least the "UPC -> Price" dataset. So download the whole dataset each night, if you can't get delta changes to work.

And if that had failed somehow, it would have been noticed immediately upon the new code roll-out.

The internet was designed to be extremely resilient to host/route losses, we've made it so reliable we assume all machines are reachable at all times.

(To be fair, apparently they "do" have this but the dataset is printed on the items and the cashiers had to enter it by hand)

Re: Hardening the registers: A cascading failure of edge induced fault tolerance

#27

Post by sokoloff »

numbsafari wrote: post #7
rjbwork wrote: post #5

Earlier quoted context omitted.

Yes. Same way they call their employees "associates". I don't quite understand the rationale, but if I had to guess, "customer" and "employee" are a bit too on the nose, and they wish to cultivate a more human-feeling relationship between the customers, employees, and corporation in the minds of the former two groups.

Not just in the minds of the former two groups, but in the minds of their staff and leadership as well. If you refer to your team members or employees as "associates" you're much more likely to treat them as equals. Similarly, if you refer to your customers as "guests", you are much more likely to treat them as such rather than simply treating them as people in your store looking to spend money. It gets to the whole…

It is for this reason that I doggedly push back on the use of "resources" when talking specifically about people; I semi-frequently correct this mis-use (IMO) of language.

If you ask "do we have enough resources to compete in segment X?" and you mean resources of all types [including people], that's fine. If you ask "could I have two additional resources on this project" and you mean exactly people, I'll speak up every time.

Re: Hardening the registers: A cascading failure of edge induced fault tolerance

#28

Post by yardstick »

nerdponx wrote: post #18
yardstick wrote: post #13

> Although Autobahn contained all the item data, the 404 responses were interpreted by the SDM Proxy as an indicator that the item was missing in Autobahn and the SDM Proxy retried the request to the central ILS API in the data centers. This is why I never design web APIs to use the HTTP status code to indicate the application response. Always embed the application response within the HTTP payload. It should be indep…

Sounds more like the server was misusing the 404 status and/or the clients were mishandling it. I am inclined to agree that for this particular usage, an "in-body" response makes sense. 404 should be reserved for when the actual HTTP endpoint is unavailable. But in REST semantics, you would only return 404 for an endpoint like /users/12345 when user 12345 doesn't exist. So the two usages line up. Returning 200 with a…

The problem is most REST services, because they are fundamentally HTTP services, are subject to how the underlying HTTP application server/proxy/middleware handles HTTP requests. Which are valid to return 404 in many more cases than where REST would allow you to return 404. In Targets case, it should have probably returned 400 Bad Client, since what the client tried to access wasn’t a REST endpoint.

The real problem with REST and HTTP is that it’s too easy to put middleware in between that doesn’t understand REST, just HTTP. As a software engineer or architect you can design the API to be perfect to your needs, but, when deployed you often lose control of how the client and server actually connect to each other. Proxies, caches, IDS, WAF, all can get in the way and don’t respect REST semantics.

Re: Hardening the registers: A cascading failure of edge induced fault tolerance

#29

Post by yardstick »

thedougd wrote: post #19
yardstick wrote: post #13

> Although Autobahn contained all the item data, the 404 responses were interpreted by the SDM Proxy as an indicator that the item was missing in Autobahn and the SDM Proxy retried the request to the central ILS API in the data centers. This is why I never design web APIs to use the HTTP status code to indicate the application response. Always embed the application response within the HTTP payload. It should be indep…

You do have to consider middlebox and client caching when you do this. Returning a 200 with a 'not found' would be cached, and that may or may not be desired for the use case.

That’s I guess an issue for cache header instructions to solve.

With 200 codes being cached you still will have problems like stale data. Wouldn’t want the Target registers using yesterdays prices for today (especially if yesterday was a super sale day like Black Friday etc).

Re: Hardening the registers: A cascading failure of edge induced fault tolerance

#30

Post by Avshalom »

Avshalom wrote: post #9
rjbwork wrote: post #5

Earlier quoted context omitted.

Yes. Same way they call their employees "associates". I don't quite understand the rationale, but if I had to guess, "customer" and "employee" are a bit too on the nose, and they wish to cultivate a more human-feeling relationship between the customers, employees, and corporation in the minds of the former two groups.

Walmart has associates, Target has team members.

Some real fun horseshit is that the stockers are "designated business owners" and cashier's are "guest advocates"
Post reply on HN