Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Hardening the registers: A cascading failure of edge induced fault tolerance

tech.target.com

Post reply on HN
Print view View original

11–20 of 80 posts

Re: Hardening the registers: A cascading failure of edge induced fault tolerance

#11

Post by formerkrogemp »

rjbwork wrote: post #5
InCityDreams wrote: post #3

I presume "guest" means "customer"?

Yes. Same way they call their employees "associates". I don't quite understand the rationale, but if I had to guess, "customer" and "employee" are a bit too on the nose, and they wish to cultivate a more human-feeling relationship between the customers, employees, and corporation in the minds of the former two groups.

Ugh, some retailers still call their employees "partners." Kroger would write that every check was 'brought to you by customers' on every paper and digital pay stub. The rosy language is always used to obfuscate the exploitation going on. It's fascinating to see Target slightly improve security over the years after multiple hacks and problems with register security.

Re: Hardening the registers: A cascading failure of edge induced fault tolerance

#12

Post by at_a_remove »

I don't work at that level, or even want to, but I did detect a dark pattern that I often complain about, but have never managed to get people to pay attention to: do not collect data unless you have attached to it a decision with two or more distinct outcomes based on that data.

Re: Hardening the registers: A cascading failure of edge induced fault tolerance

#13

Post by yardstick »

> Although Autobahn contained all the item data, the 404 responses were interpreted by the SDM Proxy as an indicator that the item was missing in Autobahn and the SDM Proxy retried the request to the central ILS API in the data centers.

This is why I never design web APIs to use the HTTP status code to indicate the application response. Always embed the application response within the HTTP payload. It should be independent of the transport mechanism. I’m ok with it not being a proper REST/RESTful service.

{ “status” : 1000, “message” : “Item not found” }

And intentionally don’t use the same status numbers as http (Ie don’t use 404 as not found, because someone will mix them up!)

Re: Hardening the registers: A cascading failure of edge induced fault tolerance

#14

Post by jabart »

haroldl wrote: post #10

This was really interesting both in exploring the architecture of a retail system and looking at how systems fail. Better to read about it and learn than to live it. I'd call it a 4 hour outage because the initial "recovery" was a result of cashiers manually typing in prices for items. Then when load decreased and they discovered that scanning items worked again the problem came right back. Maybe returning 404 for bo…

204 no content is an underused http status. 404 should be monitored as an error, 204 as, well no content available. If a status code has two responsibilities that will be a monitoring issue waiting to happen.

Re: Hardening the registers: A cascading failure of edge induced fault tolerance

#15

Post by SilasX »

rjbwork wrote: post #5
InCityDreams wrote: post #3

I presume "guest" means "customer"?

Yes. Same way they call their employees "associates". I don't quite understand the rationale, but if I had to guess, "customer" and "employee" are a bit too on the nose, and they wish to cultivate a more human-feeling relationship between the customers, employees, and corporation in the minds of the former two groups.

I'm not necessarily against that in general, but if it's a technical article for a technical audience, which this looks to be intended as, they really need to drop the marketing jargon.

Re: Hardening the registers: A cascading failure of edge induced fault tolerance

#16

Post by aftbit »

Why don't the ILS services have their own cache in front of them? Supporting a per-store cache already requires good discipline on timeouts and invalidation, so adding an additional caching layer in the datacenter between the inbound requests and ILS itself seems like it would provide for a cheap extra layer of scalability in case the per-store caches become unavailable.

Re: Hardening the registers: A cascading failure of edge induced fault tolerance

#17

Post by ngc248 »

joedissmeyer wrote: post #4

Glad to see that a big US retailer like Target is using the same types of "de-facto" observability tools that I've been using for a while at all of my various employers over the last 5+ years - which are Grafana, Prometheus and Elastic Stack (specifically the Kibana UI for the logging analysis screenshot).

>>> Grafana, Prometheus and Elastic Stack

Those 3 have almost become the industry standard for observability. Everywhere I have worked have used the same and it is almost a no-brainer.

Re: Hardening the registers: A cascading failure of edge induced fault tolerance

#18

Post by nerdponx »

yardstick wrote: post #13

> Although Autobahn contained all the item data, the 404 responses were interpreted by the SDM Proxy as an indicator that the item was missing in Autobahn and the SDM Proxy retried the request to the central ILS API in the data centers. This is why I never design web APIs to use the HTTP status code to indicate the application response. Always embed the application response within the HTTP payload. It should be indep…

Sounds more like the server was misusing the 404 status and/or the clients were mishandling it.

I am inclined to agree that for this particular usage, an "in-body" response makes sense. 404 should be reserved for when the actual HTTP endpoint is unavailable. But in REST semantics, you would only return 404 for an endpoint like /users/12345 when user 12345 doesn't exist. So the two usages line up. Returning 200 with a body that says "user 12345 does not exist" makes a lot less sense to me.

A good example of overdoing it is when GraphQL servers return a 200 HTTP response that contains nothing but an error message, instead of returning a suitable HTTP status like 400.

Re: Hardening the registers: A cascading failure of edge induced fault tolerance

#19

Post by thedougd »

yardstick wrote: post #13

> Although Autobahn contained all the item data, the 404 responses were interpreted by the SDM Proxy as an indicator that the item was missing in Autobahn and the SDM Proxy retried the request to the central ILS API in the data centers. This is why I never design web APIs to use the HTTP status code to indicate the application response. Always embed the application response within the HTTP payload. It should be indep…

You do have to consider middlebox and client caching when you do this. Returning a 200 with a 'not found' would be cached, and that may or may not be desired for the use case.

Re: Hardening the registers: A cascading failure of edge induced fault tolerance

#20

Post by TrueGeek »

yardstick wrote: post #13

> Although Autobahn contained all the item data, the 404 responses were interpreted by the SDM Proxy as an indicator that the item was missing in Autobahn and the SDM Proxy retried the request to the central ILS API in the data centers. This is why I never design web APIs to use the HTTP status code to indicate the application response. Always embed the application response within the HTTP payload. It should be indep…

I think the important thing is to be consistent. I worked with a microservice system for years (as a front end engineer) where some teams would use 404 to indicate record not found, some teams would use your system, and a couple teams sent back the response in the header! Of the teams using your system it would quite an ordeal to find out the meaning of "status: 1000", especially if the system was 10+ years old and the original team no longer around.
Post reply on HN