Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Hardening the registers: A cascading failure of edge induced fault tolerance

tech.target.com

Post reply on HN
Print view View original

61–70 of 80 posts

Re: Hardening the registers: A cascading failure of edge induced fault tolerance

#61

Post by hinkley »

yardstick wrote: post #28
nerdponx wrote: post #18

Earlier quoted context omitted.

Sounds more like the server was misusing the 404 status and/or the clients were mishandling it. I am inclined to agree that for this particular usage, an "in-body" response makes sense. 404 should be reserved for when the actual HTTP endpoint is unavailable. But in REST semantics, you would only return 404 for an endpoint like /users/12345 when user 12345 doesn't exist. So the two usages line up. Returning 200 with a…

The problem is most REST services, because they are fundamentally HTTP services, are subject to how the underlying HTTP application server/proxy/middleware handles HTTP requests. Which are valid to return 404 in many more cases than where REST would allow you to return 404. In Targets case, it should have probably returned 400 Bad Client, since what the client tried to access wasn’t a REST endpoint. The real problem…

As someone who tries to steer teams toward designing for middleboxes up front instead of trying to spackle caching into the system once it's too baroque to implement properly, I find that a little time up front avoids a lot of pain and anger later on. 400 bad request is a much better way to signal that there is no REST endpoint for that request versus there is one but we didn't find any data.

Also worrying about cache expiry is often a red herring. It's not fixing a problem, unless the problem is that the Product Owner keeps noticing that our code doesn't actually work as advertised. If you can generate a useful etag for a response, you can add client or middlebox caching any time it becomes useful, or take it away when it doesn't. But if you can't generate a useful etag, then any bespoke caching mechanism you build is unsound, because correct etag and correct cache invalidation are isomorphic.

I'd rather know we're on the road to unsound sooner rather than after customers rely on a bunch of misfeatures.

Re: Hardening the registers: A cascading failure of edge induced fault tolerance

#62

Post by barbecue_sauce »

EvanAnderson wrote: post #50
InCityDreams wrote: post #3

I presume "guest" means "customer"?

My wife worked for Darden Restarurants for awhile and corporate training materials always referred to customers as "guests", too. On one level I suppose it's just silly terminology, but it grates with me. I guess it's supposed to imply some kind of familiar relationship, free of the gauche trappings of economics. To me a customer demands more attention than a "guest". It shocks me how many people don't recognize that…

The history of food service goes hand-in-hand with the hospitality industry, so referring to a customer as a "guest" is very traditional and common amongst almost all restaurants.

Re: Hardening the registers: A cascading failure of edge induced fault tolerance

#63

Post by [deleted] »

henry700 wrote: post #59
a-dub wrote: post #45

Earlier quoted context omitted.

also maybe counting any sort of retry or fallback as a trackable and alertable failure. auto/silent fallbacks seem like clever ways of avoiding beeps and remaining resilient against failures in supporting systems, but in practice it they always tend to just cover up real issues until it's too late. i think the ideal is to have a nice easily included retry library that includes reporting/alerting, configurable backoff…

Yeah, this. If they just had a custom counter for "ILS datacenter fetch" retries after receiving a 404 from the local cache and that spiked to, say, 50% of the "every scan counter" value, something would already be seriously wrong: How does the local Target store cache have less than 50% of the needed data in it??

[deleted]

Re: Hardening the registers: A cascading failure of edge induced fault tolerance

#65

Post by hn_throwaway_99 »

jaywalk wrote: post #23
yardstick wrote: post #13

> Although Autobahn contained all the item data, the 404 responses were interpreted by the SDM Proxy as an indicator that the item was missing in Autobahn and the SDM Proxy retried the request to the central ILS API in the data centers. This is why I never design web APIs to use the HTTP status code to indicate the application response. Always embed the application response within the HTTP payload. It should be indep…

I think a better solution would be to use the 404 code but include a body with the detailed error. That way the response to an invalid URL looks different from an actual item not being found.

But many clients don't even look at the body if the response is a 404.

Re: Hardening the registers: A cascading failure of edge induced fault tolerance

#66

Post by nerdponx »

hinkley wrote: post #61
yardstick wrote: post #28

Earlier quoted context omitted.

The problem is most REST services, because they are fundamentally HTTP services, are subject to how the underlying HTTP application server/proxy/middleware handles HTTP requests. Which are valid to return 404 in many more cases than where REST would allow you to return 404. In Targets case, it should have probably returned 400 Bad Client, since what the client tried to access wasn’t a REST endpoint. The real problem…

As someone who tries to steer teams toward designing for middleboxes up front instead of trying to spackle caching into the system once it's too baroque to implement properly, I find that a little time up front avoids a lot of pain and anger later on. 400 bad request is a much better way to signal that there is no REST endpoint for that request versus there is one but we didn't find any data. Also worrying about cach…

Do you feel like it's a mistake to conflate URL paths with resources?

Because it sounds like you take issue with the conflation of "404 because user 12345 doesn't exist" and "404 because this url path is malformed", and (if I understand your post) are suggesting that the latter should be 400 so as to allow the former to be 404.

What about instead using 204 to signal "does not exist in database but request is otherwise valid"?

Re: Hardening the registers: A cascading failure of edge induced fault tolerance

#67

Post by kevan »

prithvi24 wrote: post #55

Target has 250k SKUs total - why is their inventory system so complicated? Why the hybrid on-prem store + data center cloud model - isn’t it easier if there is one source of truth? Seems like it would reduce the need for even dealing with all this eventually consistence cache sycning and whatnot I ofc don’t know what I dont know, but super curious if anyone has insight into why such a complex system is required Also,…

Not being able to check out customers is a really bad customer experience. It's a double whammy of wasting their time and they don't even get what they needed so it's worth investing to make that less likely. Things are probably better now but when I worked at a Sears our network connection to HQ wasn't reliable enough to depend on completely for checkout operations.

Re: Hardening the registers: A cascading failure of edge induced fault tolerance

#68

Post by hn_throwaway_99 »

yardstick wrote: post #13

> Although Autobahn contained all the item data, the 404 responses were interpreted by the SDM Proxy as an indicator that the item was missing in Autobahn and the SDM Proxy retried the request to the central ILS API in the data centers. This is why I never design web APIs to use the HTTP status code to indicate the application response. Always embed the application response within the HTTP payload. It should be indep…

This is one (of many) reasons why I prefer GraphQL. The error codes are clearly defined and contained in the body of a 200 response.

Re: Hardening the registers: A cascading failure of edge induced fault tolerance

#69

Post by Avshalom »

londons_explore wrote: post #51

I'd like to see staff training for major outages in retail like this. For example, if the shop loses power, do they have the ability to sell goods still? One approach is to let staff members estimate the value of goods - for example at the register, the staff member looks at the cart contents, estimates that it's about $120 worth of goods, charges the customer $120, and hand writes a receipt saying "$120 of goods sol…

>>For example, if the shop loses power, do they have the ability to sell goods still

To some degree yes, we can check people out with a handheld (which has swappable batteries) and the self check registers are on the emergency power circuit.

Couple years ago when the system went down nationwide we just told people to put their name on their cart and we gave them 10% off if the came back the next day.

Re: Hardening the registers: A cascading failure of edge induced fault tolerance

#70

Post by EricE »

"It’s not enough to implement redundant systems and failovers, we must monitor and alert when those systems are being exercised."

My air conditioner in my house has a secondary drain pan under it. The outlet for that drain pan is right above a main window outside. If the primary condensate drain gets plugged/fails and the water overflows into the backup pan there would be a stream of water in front of a window that shouldn't otherwise be there. They want you to be able to readily notice it as you are now at risk for significant water damage if that secondary drain manages to plug up too.

Always something worth considering when designing any system - how to make it fail in a way that is noticeable!

Post reply on HN