Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Never trust a system that seems to be working

twitter.com

Post reply on HN
Print view View original

151–160 of 376 posts

Re: Never trust a system that seems to be working

#151

Post by Karellen »

pengstrom wrote: post #48
Karellen wrote: post #10

One reply tweet notes "a complex system always operates in a failure state", but to find more discussion on this point it's worth noting that this is a restatement of: > "The Fundamental Failure-Mode Theorem (F.F.T.): complex systems usually operate in a failure mode." -- John Gall, General Systemantics (aka Systemantics , aka The Systems Bible ), 1977 https://en.wikipedia.org/wiki/Systemantics > Searching for "funda…

The older I get and the more systems I encounter, the more I'm convinced everything is in a state of failing. The question is only how long before it becomes noticable.

One of the consequences of this is when something goes very wrong, and the people involved say it really shouldn't have because something like 4 separate things have to fail at the same time for that to have happened.

What they don't realise is that, thanks to the FFT, 2 of those 4 things actually failed over a year ago and the system has been continuing to work ever since without anyone ever noticing. One of the other things actually fails intermittently a couple of times per month, and while a couple of people have noticed that and reported it, either it's not been investigated at all, or when it was investigated everything looked fine and the report was put down as an unexplained glitch. All that was actually needed was for the 4th thing to fail, to cause this "can't happen" failure mode to suddenly ruin everyone's day.

Re: Never trust a system that seems to be working

#152

Post by outworlder »

kbenson wrote: post #70
pengstrom wrote: post #48

Earlier quoted context omitted.

The older I get and the more systems I encounter, the more I'm convinced everything is in a state of failing. The question is only how long before it becomes noticable.

Experienced sysadmins see the occasional chatter of problems in logs and emails as the pulse of the system, the EKG hooked up to the patient letting you know how things are going. The right level of logging is usually enough to be annoying but not so annoying you need to reduce it. That means you're hopefully thorough enough in the logging that you're reporting what you care about and the things you see are the the t…

> because that only scales so far (but surprisingly far for diligent admins)

Surprisingly far. We log terabytes of data a day, most of it is unnecessary. I've been trying to tell people that it is not a good idea but so far I've had limited success. It's easy for people to just "printf" everything (we are lucky if logging libraries are used with proper levels).

Right now, we have the most expensive 'add' operation in history. K8s pod logs some data. That is eventually noticed by fluentd, which is watching the filesystem. Fluentd sends to our logging servers over the network. That goes to kafka, then is shipped to the system that actually does the indexing. And then, there's a query running, which will then count the number of times that string appeared and updates a counter.

All of which could have been avoided if the apps just added +1 to a counter, and exposed that as /metrics so Prometheus could scrape.

Re: Never trust a system that seems to be working

#153

Post by outworlder »

ben_w wrote: post #96

Reminds me of Jurassic Park (book, I think?); nobody realised that dinosaurs were loose and breeding, because the system was doing: if dinosaurs and nobody anticipated dinosaurs > expected

That was one of the system's invariants.

assert (dinosaurs <= expected)

Re: Never trust a system that seems to be working

#156

Post by mananaysiempre »

electroly wrote: post #43

This is why 4-20 mA is a common signaling standard in industrial automation. 4 mA means zero and 20 mA means one. 0 mA means broken transmitter !

Also why a standard keyboard has a Break key: used to be that a telegraph station would have a bell that rang if it detected a broken connection (a “break condition”), then people figured that deliberately inducing a break was a handy way to alert the remote operator. I think the first automated telegraph systems actually used current-loop signalling, but I’m not sure about the first practical ones; in any case, the concept carried over into modern (well, 70s-vintage) asynchronous lines—most UARTs nowadays have a break interrupt that you can enable.

Re: Never trust a system that seems to be working

#157

Post by wongarsu »

MBCook wrote: post #93
usrbinenv wrote: post #88

A bit of an off-topic: I only ever saw white traffic lights in the US and Canada (specifically, pedestrian traffic lights) and they confused me a lot at first - the rest of the world uses green. The icons displayed (hands) are confusing too: the rest of the world uses an icon of a man walking or standing.

I’ve never seen a white traffic light, but pedestrian crossing lights are often white/red still.

Are you suggesting I'm doing worse on Google Captchas because I consider pedestrian crossing lights as a form of traffic lights but Americans don't?

Re: Never trust a system that seems to be working

#158

Post by lawrenceyan »

In the self-driving context, this reminds me strongly of Tesla Autopilot. Good enough to work most of the time, but likely would end up in greater overall injuries/accidents per mile if actually enabled at scale.

Waymo, Cruise, Aurora, and others are doing it the right way.

Re: Never trust a system that seems to be working

#159

Post by ianferrel »

usrbinenv wrote: post #147
malloci wrote: post #128

Earlier quoted context omitted.

The hand is to mimic that of a traffic cop. When they want you to stop, they hold up their hand to indicate that you need to stop. It used to be that you had to know english in order to understand them as well...many of the signs used to use the words "walk" and "don't walk"

I guess my point is that it was still confusing. It took me a day or two to acquire a reflex. It obviously doesn't matter in the long run - you just get used to it - but I wonder why would US & Canada go a different way with regards to so many small things (think turn signals on cars being red for american cars as another example)? What would be the rationale? It's as if the western hemisphere was completely cut off…

>I wonder why would US & Canada go a different way with regards to so many small things (think turn signals on cars being red for american cars as another example)?

In many cases the US were the first to mass-manufacture things to some kind of standard, so the question is really: why did the rest of the world choose a different standard. Often they have a good reason (better design, maybe), but the US also has a good reason for not switching (inertia, existing tooling, people understand the existing standard).

The US was the first place that traffic light and crossing signals were installed. The reason they didn't go with the standard the rest of the world uses is that it didn't exist.

Re: Never trust a system that seems to be working

#160

Post by speeder »

outworlder wrote: post #103
Kiro wrote: post #49

Earlier quoted context omitted.

This is a mod that seems extremely boring and tedious. The base game is not like this.

> This is a mod that seems extremely boring and tedious. It doesn't have to be, though. I'm currently playing Space Exploration + Krastorio 2. Krastorio modifies the early/mid game, Space Exploration meddles a little with the early game, but most of it is late game. There are a few more inputs required for things so they can get more complicated. And with Krastorio the early game is extended (you have "burner" versio…

Burner things is from space exploration too. Krastorio doesn't change the very early game.
Post reply on HN