Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Hardening the registers: A cascading failure of edge induced fault tolerance

tech.target.com

Post reply on HN
Print view View original

51–60 of 80 posts

Re: Hardening the registers: A cascading failure of edge induced fault tolerance

#51

Post by londons_explore »

I'd like to see staff training for major outages in retail like this.

For example, if the shop loses power, do they have the ability to sell goods still?

One approach is to let staff members estimate the value of goods - for example at the register, the staff member looks at the cart contents, estimates that it's about $120 worth of goods, charges the customer $120, and hand writes a receipt saying "$120 of goods sold, Date, store name, signature". The staff member then uses a phone to photograph the cart and the receipt.

At the end of the shift, the shaff member drops all the photos into a big store wide Dropbox account, that the accounts department can use to pay taxes.

You'd probably want to practice this process ahead of time with every staff member.

I imagine it might actually be a good process to use on very busy days too - it is probably quicker than scanning every item at the register.

Re: Hardening the registers: A cascading failure of edge induced fault tolerance

#52

Post by bombcar »

jaywalk wrote: post #37
bombcar wrote: post #26

Earlier quoted context omitted.

The total size of all item data for a store like Target can't be much more than what, a few gigabytes? Or at least the "UPC -> Price" dataset. So download the whole dataset each night, if you can't get delta changes to work. And if that had failed somehow, it would have been noticed immediately upon the new code roll-out. The internet was designed to be extremely resilient to host/route losses, we've made it so relia…

Their system was basically designed the way you're saying, with a fallback to grab the data from the central location if it's missing locally. What you're asking for is the same system without a fallback, which doesn't make any sense.

The fallback was the problem - design it without it, or with a manual window that pops up saying "ITEM NOT FOUND, QUERY TARGET ORACLE" or something, and the fault wouldn't have taken down the whole company.

If suddenly every cashier is being forced to hit OK on every item, people would hear about it immediately from the test rollout instead of when it hit everything (of course, assuming you have good methods for detecting things like this and don't just completely ignore associates' complaints).

Re: Hardening the registers: A cascading failure of edge induced fault tolerance

#53

Post by reaperducer »

jkaptur wrote: post #49

It's very interesting that by building a system that's more resilient and reliable: > high profile processes (such as POS) implement their own fallback processes to handle the possibility of issues with the SDM system in store. In the case of item data, the POS software on each register is capable of bypassing the SDM Proxy and retrying its request directly to the ILS API in the data centers. ... the system as a whol…

I find all of this fascinating.

A few years ago, the guys who built Chick-fil-a's POS fog were on HN talking about their fault-tolerance and transaction queueing. It was quite interesting.

There's a lot that you can learn from high-volume POS system design that applies to just bog-standard every day programming.

Re: Hardening the registers: A cascading failure of edge induced fault tolerance

#54

Post by reaperducer »

throwaway894345 wrote: post #48
tornato7 wrote: post #24

Earlier quoted context omitted.

Reminds me of an online ordering app at my university - when the API went down one day the app started reporting to everyone that the wait time for their food was “503 minutes”

That's just awful programming, and the parent's suggestion wouldn't have addressed that. It would have instead resulted in "1000 minutes" or whatever their status code was.

That's just awful programming, and the parent's suggestion wouldn't have addressed that.

It would have if the API used the HTTP status number to indicate the number of minutes.

I've seen crazier things in web APIs.

Re: Hardening the registers: A cascading failure of edge induced fault tolerance

#55

Post by prithvi24 »

Target has 250k SKUs total - why is their inventory system so complicated? Why the hybrid on-prem store + data center cloud model - isn’t it easier if there is one source of truth? Seems like it would reduce the need for even dealing with all this eventually consistence cache sycning and whatnot

I ofc don’t know what I dont know, but super curious if anyone has insight into why such a complex system is required

Also, if this microservice is used for brick and mortgage mortar, can’t imagine more than a couple hundred per second? ( 2000 stores, 5 registers a store - and humans manually scanning items ) - why did that overload the micro service (guessing it wasn’t an endless exponential backoff)

Re: Hardening the registers: A cascading failure of edge induced fault tolerance

#56

Post by bombcar »

jkaptur wrote: post #49

It's very interesting that by building a system that's more resilient and reliable: > high profile processes (such as POS) implement their own fallback processes to handle the possibility of issues with the SDM system in store. In the case of item data, the POS software on each register is capable of bypassing the SDM Proxy and retrying its request directly to the ILS API in the data centers. ... the system as a whol…

Buried under another thread was this post: https://aws.amazon.com/builders-library/avoiding-fallback-in... which is the exact same issue - a cache miss was backed up by a direct query and it took down all of Amazon trying to display shipping times.

grok say complexity bad

Fallback is not always necessary (sometimes it is, you can't just say "whelp the engines on this plane went out, time to die") but when you have a fallback system you should think about why you have it and how bad it is to fail, and if it could be worse to succeed.

Re: Hardening the registers: A cascading failure of edge induced fault tolerance

#57

Post by lalaland1125 »

prithvi24 wrote: post #55

Target has 250k SKUs total - why is their inventory system so complicated? Why the hybrid on-prem store + data center cloud model - isn’t it easier if there is one source of truth? Seems like it would reduce the need for even dealing with all this eventually consistence cache sycning and whatnot I ofc don’t know what I dont know, but super curious if anyone has insight into why such a complex system is required Also,…

> I ofc don’t know what I dont know, but super curious if anyone has insight into why such a complex system is required

Because it's much more efficient, which allows them to use simpler tech that doesn't need to scale as well.

You are also underestimated the throughput the system needs to handle. 2000 stores * 10 registers per store * 1000 scans per register per hour = 5000 scans per second.

Re: Hardening the registers: A cascading failure of edge induced fault tolerance

#58

Post by marcosdumay »

rjbwork wrote: post #5
InCityDreams wrote: post #3

I presume "guest" means "customer"?

Yes. Same way they call their employees "associates". I don't quite understand the rationale, but if I had to guess, "customer" and "employee" are a bit too on the nose, and they wish to cultivate a more human-feeling relationship between the customers, employees, and corporation in the minds of the former two groups.

I always assumed that "associates" was created to encode the idea that people's salary was mostly commission based. But with you talking about those giant corporations that call everybody by that name, this is either anachronistic or plain wrong.

Re: Hardening the registers: A cascading failure of edge induced fault tolerance

#59

Post by henry700 »

a-dub wrote: post #45
yardstick wrote: post #13

> Although Autobahn contained all the item data, the 404 responses were interpreted by the SDM Proxy as an indicator that the item was missing in Autobahn and the SDM Proxy retried the request to the central ILS API in the data centers. This is why I never design web APIs to use the HTTP status code to indicate the application response. Always embed the application response within the HTTP payload. It should be indep…

also maybe counting any sort of retry or fallback as a trackable and alertable failure. auto/silent fallbacks seem like clever ways of avoiding beeps and remaining resilient against failures in supporting systems, but in practice it they always tend to just cover up real issues until it's too late. i think the ideal is to have a nice easily included retry library that includes reporting/alerting, configurable backoff…

Yeah, this. If they just had a custom counter for "ILS datacenter fetch" retries after receiving a 404 from the local cache and that spiked to, say, 50% of the "every scan counter" value, something would already be seriously wrong: How does the local Target store cache have less than 50% of the needed data in it??

Re: Hardening the registers: A cascading failure of edge induced fault tolerance

#60

Post by prithvi24 »

lalaland1125 wrote: post #57
prithvi24 wrote: post #55

Target has 250k SKUs total - why is their inventory system so complicated? Why the hybrid on-prem store + data center cloud model - isn’t it easier if there is one source of truth? Seems like it would reduce the need for even dealing with all this eventually consistence cache sycning and whatnot I ofc don’t know what I dont know, but super curious if anyone has insight into why such a complex system is required Also,…

> I ofc don’t know what I dont know, but super curious if anyone has insight into why such a complex system is required Because it's much more efficient, which allows them to use simpler tech that doesn't need to scale as well. You are also underestimated the throughput the system needs to handle. 2000 stores * 10 registers per store * 1000 scans per register per hour = 5000 scans per second.

I’m not sure the throughput is that high - scans take quiet a bit of time, I would doubt that a register scans an item every 3.6 seconds - don’t have data on this but would easily triple that estimate as an average (so in the hundreds)

Also , I get the simpler tech, but complexity breeds failure - if you have a hybrid on prem / cloud model, especially with only 250k skus, at that point doesn’t it make sense to keep that exclusively in the cloud.

It’s a system that scans a barcode and returns an item at its core - this is still well under the limits of using an off the shelf system like Redis behind an endpoint

Post reply on HN