Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Hardening the registers: A cascading failure of edge induced fault tolerance

tech.target.com

Post reply on HN
Print view View original

71–80 of 80 posts

Re: Hardening the registers: A cascading failure of edge induced fault tolerance

#71

Post by EricE »

prithvi24 wrote: post #60
lalaland1125 wrote: post #57

Earlier quoted context omitted.

> I ofc don’t know what I dont know, but super curious if anyone has insight into why such a complex system is required Because it's much more efficient, which allows them to use simpler tech that doesn't need to scale as well. You are also underestimated the throughput the system needs to handle. 2000 stores * 10 registers per store * 1000 scans per register per hour = 5000 scans per second.

I’m not sure the throughput is that high - scans take quiet a bit of time, I would doubt that a register scans an item every 3.6 seconds - don’t have data on this but would easily triple that estimate as an average (so in the hundreds) Also , I get the simpler tech, but complexity breeds failure - if you have a hybrid on prem / cloud model, especially with only 250k skus, at that point doesn’t it make sense to keep t…

"I would doubt that a register scans an item every 3.6 seconds"

Indeed, that sounds WAY too slow for me - traffic like this is bursty. Ever try to scan five of the same thing at some self check out registers? On some it's instantaneous (an awesome customer experience) on others there are one second or more delays (horrible customer experience).

Latency = friction and friction is the ultimate deal killer.

Re: Hardening the registers: A cascading failure of edge induced fault tolerance

#72

Post by prewett »

EricE wrote: post #71
prithvi24 wrote: post #60

Earlier quoted context omitted.

I’m not sure the throughput is that high - scans take quiet a bit of time, I would doubt that a register scans an item every 3.6 seconds - don’t have data on this but would easily triple that estimate as an average (so in the hundreds) Also , I get the simpler tech, but complexity breeds failure - if you have a hybrid on prem / cloud model, especially with only 250k skus, at that point doesn’t it make sense to keep t…

"I would doubt that a register scans an item every 3.6 seconds" Indeed, that sounds WAY too slow for me - traffic like this is bursty. Ever try to scan five of the same thing at some self check out registers? On some it's instantaneous (an awesome customer experience) on others there are one second or more delays (horrible customer experience). Latency = friction and friction is the ultimate deal killer.

Walmart is the worst, it takes multiple seconds to even register that the scan worked.

Re: Hardening the registers: A cascading failure of edge induced fault tolerance

#73

Post by cptskippy »

yardstick wrote: post #28
nerdponx wrote: post #18

Earlier quoted context omitted.

Sounds more like the server was misusing the 404 status and/or the clients were mishandling it. I am inclined to agree that for this particular usage, an "in-body" response makes sense. 404 should be reserved for when the actual HTTP endpoint is unavailable. But in REST semantics, you would only return 404 for an endpoint like /users/12345 when user 12345 doesn't exist. So the two usages line up. Returning 200 with a…

The problem is most REST services, because they are fundamentally HTTP services, are subject to how the underlying HTTP application server/proxy/middleware handles HTTP requests. Which are valid to return 404 in many more cases than where REST would allow you to return 404. In Targets case, it should have probably returned 400 Bad Client, since what the client tried to access wasn’t a REST endpoint. The real problem…

> In Targets case, it should have probably returned 400 Bad Client, since what the client tried to access wasn’t a REST endpoint.

The trouble with that approach is it puts the onus on the Server to response accordingly. In the case of a Client misconfiguration you might point the Client at a valid HTTP Server, just not the one you anticipated.

I agree with nerdponx that they were miss using 404. Instead they should have used a different HTTP Response Status Code to indicate something was removed from the system. Perhaps 410 Gone or 417 Expectation Failed.

Re: Hardening the registers: A cascading failure of edge induced fault tolerance

#74

Post by jaywalk »

hn_throwaway_99 wrote: post #65
jaywalk wrote: post #23

Earlier quoted context omitted.

I think a better solution would be to use the 404 code but include a body with the detailed error. That way the response to an invalid URL looks different from an actual item not being found.

But many clients don't even look at the body if the response is a 404.

True, but we're talking about fully internal systems here, where one party is in control of both the client and server.

Re: Hardening the registers: A cascading failure of edge induced fault tolerance

#75

Post by layer8 »

cptskippy wrote: post #73
yardstick wrote: post #28

Earlier quoted context omitted.

The problem is most REST services, because they are fundamentally HTTP services, are subject to how the underlying HTTP application server/proxy/middleware handles HTTP requests. Which are valid to return 404 in many more cases than where REST would allow you to return 404. In Targets case, it should have probably returned 400 Bad Client, since what the client tried to access wasn’t a REST endpoint. The real problem…

> In Targets case, it should have probably returned 400 Bad Client, since what the client tried to access wasn’t a REST endpoint. The trouble with that approach is it puts the onus on the Server to response accordingly. In the case of a Client misconfiguration you might point the Client at a valid HTTP Server, just not the one you anticipated. I agree with nerdponx that they were miss using 404. Instead they should h…

417 is specifically for use with the Expect header, so would be a misuse as well. 410 would be fine, but is technically a sub-case of 404, so if 410 is fine 404 should arguably be too.

Re: Hardening the registers: A cascading failure of edge induced fault tolerance

#76

Post by layer8 »

joeframbach wrote: post #41
yardstick wrote: post #13

> Although Autobahn contained all the item data, the 404 responses were interpreted by the SDM Proxy as an indicator that the item was missing in Autobahn and the SDM Proxy retried the request to the central ILS API in the data centers. This is why I never design web APIs to use the HTTP status code to indicate the application response. Always embed the application response within the HTTP payload. It should be indep…

The proper response should have been 400 Bad Request, not 404 Not Found. Because the client's request contained bad request data. Not that the item wasn't found.

If you place a standard web server in default configuration (serving an empty directory tree of static files) on that host, it will also return 404 for any GET request. By that I mean, the client has in principle to expect that 404 means “I asked the wrong server” or “the server is misconfigured” or “my endpoint is gone”.

Re: Hardening the registers: A cascading failure of edge induced fault tolerance

#77

Post by magicalhippo »

jabart wrote: post #14
haroldl wrote: post #10

This was really interesting both in exploring the architecture of a retail system and looking at how systems fail. Better to read about it and learn than to live it. I'd call it a 4 hour outage because the initial "recovery" was a result of cashiers manually typing in prices for items. Then when load decreased and they discovered that scanning items worked again the problem came right back. Maybe returning 404 for bo…

204 no content is an underused http status. 404 should be monitored as an error, 204 as, well no content available. If a status code has two responsibilities that will be a monitoring issue waiting to happen.

So your suggestion is that if I have a /invoice endpoint, a "GET /invoice/abc123" should return 204 if it's an invalid/non-existing invoice number?

Seems reasonable.

Re: Hardening the registers: A cascading failure of edge induced fault tolerance

#79

Post by hinkley »

nerdponx wrote: post #66
hinkley wrote: post #61

Earlier quoted context omitted.

As someone who tries to steer teams toward designing for middleboxes up front instead of trying to spackle caching into the system once it's too baroque to implement properly, I find that a little time up front avoids a lot of pain and anger later on. 400 bad request is a much better way to signal that there is no REST endpoint for that request versus there is one but we didn't find any data. Also worrying about cach…

Do you feel like it's a mistake to conflate URL paths with resources? Because it sounds like you take issue with the conflation of "404 because user 12345 doesn't exist" and "404 because this url path is malformed", and (if I understand your post) are suggesting that the latter should be 400 so as to allow the former to be 404. What about instead using 204 to signal "does not exist in database but request is otherwis…

Well I think 404 is only the first problem you have to solve and people don't even do that very well, then declare success and move on. I've seen so many developers try to reinvent things that are in the HTTP 1.0 specification.

I've definitely had to deal with problems of people returning 200 not found and those leaking up into higher layers of the system, and I agree with the people who don't like that solution. 204 is probably better, but I've seen too much code that looks for 200On one project, I put the kibosh on someone trying to negotiate clock skew detection between the client and server. I don't recall what they were going to do about it but I pointed out that information is already in the request headers, and has been from the start. So if your timing decisions are tied to an HTTP request, you don't need to do anything out of band to negotiate a common notion of time.

I don't recall when I stopped seeing timestamps of January 2, 1970, but I know I saw enough of them that it stopped being a novelty and I stopped looking for them. In 1995 when they were still working on the spec, nobody used NTP yet, and many, many people had a dead or dying CR2032 battery on their motherboard and either hadn't noticed or didn't know where to buy a replacement. The dying ones were the worst because sometimes they would hold the time but not increment it properly, so the clock skew was proportional to how many hours a day they turned the machine off. "Expires on March 1" didn't mean anything, unless you knew that the server thought it was February 28th. I suspect Tim was thinking of time zones between scientists in Switzerland, France, England and Germany, but it worked really well for crap hardware too.

Re: Hardening the registers: A cascading failure of edge induced fault tolerance

#80

Post by kerbs »

marcosdumay wrote: post #46
kerbs wrote: post #43

Earlier quoted context omitted.

Targets mobile apps were down over Black Friday many years back for a very similar reason – logic done on status codes. A 403 in the API had a very specific meaning, and when the proxy layer started returning 403s everyone had a really bad time. (That was a long day)

> A 403 in the API had a very specific meaning And that meaning wasn't "you are authenticated as a user that can not access this resource"?

It meant the client was expected to then make a request to refresh their session token.

Because of the middle layer sending a 403 instead of the API, clients would request refresh tokens in an infinite loop.

Post reply on HN