A bit of an off-topic: I only ever saw white traffic lights in the US and Canada (specifically, pedestrian traffic lights) and they confused me a lot at first - the rest of the world uses green. The icons displayed (hands) are confusing too: the rest of the world uses an icon of a man walking or standing.
I may be biased because I am used to the hand but I find the walk and hand shapes are very distinctive compared to the walk and stand.
Train air brakes are similar in failing safely for the same written-in-blood reasons: https://en.wikipedia.org/wiki/Railway_air_brake
Some military trucks are designed the opposite way on purpose - because they’d rather have no brakes and be able to move than be stuck on the battlefield.
Not in the US. US trucks are built using standard truck parts, including off-the-shelf air brakes that fail safe.
One reply tweet notes "a complex system always operates in a failure state", but to find more discussion on this point it's worth noting that this is a restatement of: > "The Fundamental Failure-Mode Theorem (F.F.T.): complex systems usually operate in a failure mode." -- John Gall, General Systemantics (aka Systemantics , aka The Systems Bible ), 1977 https://en.wikipedia.org/wiki/Systemantics > Searching for "funda…
Thank you for introducing me to this rabbit hole. Here's a NYTimes article from 1976 -- note, many issues with the OCR, but still interesting and entertainingly written: https://www.nytimes.com/1976/12/26/archives/why-nothing-work...
Same author wrote an entire book, The Systems Bible, on the topic.
There are two good lessons here: 1) Robust system design involves identifying the parts of your system that are mission-critical and always monitoring them. NASA missions have great automation and a 24/7-staffed mission control. 2) If a system failure can result in massive secondary damage, isolate that system. Warehouses receiving orbital payloads should probably be nice and far away from the base you care about.
> Warehouses receiving orbital payloads should probably be nice and far away from the base you care about.
As mentioned elsewhere in the thread, shades of 'The Moon is a Harsh Mistress' by Heinlein.
My favorite is, after fixing a system in prod, asking "how did that possibly work in the first place?"
I've totally asked that many, many times. But sometimes the answer is actually quite simple: no one was ever paying attention to the outputs of that piece of the system.
well, I'm the sort of person who digs around logs of ostensibly working systems just to find problems, which has been very useful in identifying long-tail problems (like, 1% of your network cables are blipping but the application just runs slower instead of failing).
Was thinking the same. Once you exceed like 5 tweets in a thread, it becomes a small article and you would've been better off just writing an article in the first place. If you check the thread reader below you can see just how unstructured and incoherent it is compared with a simple article.
If you're genuinely curious:
> I have ADHD: it makes writing blogs hard but Twitter easy. So it gets quickly tiring that every time I'm posted there [to HN] that the comments immediately go "why isn't this a blog? Twitter is a bad site for this kind of writing and I hate it blarg"
If your code has side effects, it only ever seems to be working. ---- > the newbie says "aww, why isn't it working?" > the intermediate says "yay, it's working!" > the expert goes "hmm, why is it working?" I've also seen this as My thing isn't working and I change X and now I get error J instead of error K, so now I have to change X back because it's still not working. Where K is a "better" error than J, such as K =…
Similarly, the joke that the engineer’s second worst nightmare is, “it doesn’t work but it should”, while their worst nightmare is, “it works but it shouldn’t”.
Tests that output booleans (pass/fail) are an antipattern. Tests and their dashboards must distinguish “the testing system worked and the test failed” from “the testing system failed”.
I dunno, I think regression tests are inherently boolean. Either it passed or it didn't. The code path that the system takes to get to a green light or not is arbitrary. If you start using an enum to return the result of the test, folk tend to add more and more enums over time until it's overly complicated. Processes return an 8 bit integer, but 99% of the time all anyone checks is whether it was 0. There's certainly…
If a regression test fails, you alert the programmer who kicked it off and maybe roll back the checkin.
If the regression testing system has an error while running a test, you page the oncall SRE and maybe try to rerun it. You certainly don’t roll back any checkins.