Live data from Hacker News

Hardening the registers: A cascading failure of edge induced fault tolerance

tech.target.com

31–40 of 80 posts

Re: Hardening the registers: A cascading failure of edge induced fault tolerance

#31

I don't work at that level, or even want to, but I did detect a dark pattern that I often complain about, but have never managed to get people to pay attention to: do not collect data unless you have attached to it a decision with two or more distinct outcomes based on that data.

Do you mean "don't collect metrics unless you monitor them and have alerts" or "don't collect data on products, customers, sales..."

Re: Hardening the registers: A cascading failure of edge induced fault tolerance

#32

> Although Autobahn contained all the item data, the 404 responses were interpreted by the SDM Proxy as an indicator that the item was missing in Autobahn and the SDM Proxy retried the request to the central ILS API in the data centers. This is why I never design web APIs to use the HTTP status code to indicate the application response. Always embed the application response within the HTTP payload. It should be indep…

[deleted]

Re: Hardening the registers: A cascading failure of edge induced fault tolerance

#33

I don't work at that level, or even want to, but I did detect a dark pattern that I often complain about, but have never managed to get people to pay attention to: do not collect data unless you have attached to it a decision with two or more distinct outcomes based on that data.

Can you give an example of what you mean?

So when I ran the university website, the homepage naturally had links to other sites. One guy had this inflated sense of importance. If there weren't a lot of clicks over to his site, we should MAKE THE LINK BIGGER because people weren't seeing it. If clicks to his site went up, we should MAKE THE LINK BIGGER because it is that important. His flowchart had only one distinct outcome: MAKE THE LINK BIGGER.

All of the effort that went into collecting the information was for nought, because the outcome was always the same. That was collection with a flowchart, but without two or more distinct outcomes.

A second example would be search engine logs. Nobody wanted to make decisions on them, but "we could always trawl them for data later." A decade on, this had never occurred. That was collection with no flow chart. Offloading the logs, parsing them out, making the data available, week after week, month after month, year after year. Wasted effort.

So part of it is "don't waste effort," but the other part is, if there is decent information to collect, you should be doing something with it.

Re: Hardening the registers: A cascading failure of edge induced fault tolerance

#34

> Although Autobahn contained all the item data, the 404 responses were interpreted by the SDM Proxy as an indicator that the item was missing in Autobahn and the SDM Proxy retried the request to the central ILS API in the data centers. This is why I never design web APIs to use the HTTP status code to indicate the application response. Always embed the application response within the HTTP payload. It should be indep…

I think the important thing is to be consistent. I worked with a microservice system for years (as a front end engineer) where some teams would use 404 to indicate record not found, some teams would use your system, and a couple teams sent back the response in the header! Of the teams using your system it would quite an ordeal to find out the meaning of "status: 1000", especially if the system was 10+ years old and t…

Fair points. My APIs have an associated constants classes/header files to define all these values.

So long as you have the source you are fine - and if you (or the third party/maintainer) don’t have the source then it doesn’t matter the approach because there will be bugs you can’t fix throughout the service.

Re: Hardening the registers: A cascading failure of edge induced fault tolerance

#35
post #15
post #5

Earlier quoted context omitted.

Yes. Same way they call their employees "associates". I don't quite understand the rationale, but if I had to guess, "customer" and "employee" are a bit too on the nose, and they wish to cultivate a more human-feeling relationship between the customers, employees, and corporation in the minds of the former two groups.

I'm not necessarily against that in general, but if it's a technical article for a technical audience, which this looks to be intended as, they really need to drop the marketing jargon.

Anything published by a corporation of Target's size is, at some level, marketing.

Re: Hardening the registers: A cascading failure of edge induced fault tolerance

#36

I don't work at that level, or even want to, but I did detect a dark pattern that I often complain about, but have never managed to get people to pay attention to: do not collect data unless you have attached to it a decision with two or more distinct outcomes based on that data.

Do you mean "don't collect metrics unless you monitor them and have alerts" or "don't collect data on products, customers, sales..."

Yes.

Re: Hardening the registers: A cascading failure of edge induced fault tolerance

#37
post #26
post #25

Earlier quoted context omitted.

If the local system doesn't have the item data (or thinks it doesn't have the item data, because it's looking in the wrong place) where exactly is it supposed to get the item data from if not the central system?

The total size of all item data for a store like Target can't be much more than what, a few gigabytes? Or at least the "UPC -> Price" dataset. So download the whole dataset each night, if you can't get delta changes to work. And if that had failed somehow, it would have been noticed immediately upon the new code roll-out. The internet was designed to be extremely resilient to host/route losses, we've made it so relia…

Their system was basically designed the way you're saying, with a fallback to grab the data from the central location if it's missing locally. What you're asking for is the same system without a fallback, which doesn't make any sense.

Re: Hardening the registers: A cascading failure of edge induced fault tolerance

#38
post #9

Earlier quoted context omitted.

Walmart has associates, Target has team members.

Some real fun horseshit is that the stockers are "designated business owners" and cashier's are "guest advocates"

I'll have you know, back at my high school job at Subway, employees were known as sandwich artists. So fancy!

Re: Hardening the registers: A cascading failure of edge induced fault tolerance

#39

Earlier quoted context omitted.

Sounds more like the server was misusing the 404 status and/or the clients were mishandling it. I am inclined to agree that for this particular usage, an "in-body" response makes sense. 404 should be reserved for when the actual HTTP endpoint is unavailable. But in REST semantics, you would only return 404 for an endpoint like /users/12345 when user 12345 doesn't exist. So the two usages line up. Returning 200 with a…

The problem is most REST services, because they are fundamentally HTTP services, are subject to how the underlying HTTP application server/proxy/middleware handles HTTP requests. Which are valid to return 404 in many more cases than where REST would allow you to return 404. In Targets case, it should have probably returned 400 Bad Client, since what the client tried to access wasn’t a REST endpoint. The real problem…

> The real problem with REST and HTTP is that it’s too easy to put middleware in between that doesn’t understand REST, just HTTP.

REST isn't a protocol, but a major point of REST’s direction to use the underlying protocol as specified is that if your are doing REST, middleware doesn't need to understand REST, only the underlying protocol.

And the problem here wasn't middleware that wasn't aware of REST, it was middleware that was misconfigured and requesting data from the wrong remapped URLs and relaying the responses faithfully. There was nothing failing to respect REST semantics involved.

Re: Hardening the registers: A cascading failure of edge induced fault tolerance

#40
post #5

Earlier quoted context omitted.

Yes. Same way they call their employees "associates". I don't quite understand the rationale, but if I had to guess, "customer" and "employee" are a bit too on the nose, and they wish to cultivate a more human-feeling relationship between the customers, employees, and corporation in the minds of the former two groups.

Not just in the minds of the former two groups, but in the minds of their staff and leadership as well. If you refer to your team members or employees as "associates" you're much more likely to treat them as equals. Similarly, if you refer to your customers as "guests", you are much more likely to treat them as such rather than simply treating them as people in your store looking to spend money. It gets to the whole…

Not all associates are at the same level. Some people unfamiliar with this American Business Vocabulary might jump to conclusions.

Some associates are the customers of the systems that you are responsible for and you are the customer for services other associates maintain.

Unfortunately rather than talk about the importance of respect and what happens when respect between members of groups within the organization is violated, these sorts of neurolinguistic fashions are used.

Post reply on HN