61–70 of 125 posts
Earlier quoted context omitted.
But why do you need an app where webpage is more than enough?
Because it cost like $10 and 5 minutes, could be set up by non-web-plumbers, and was pretty out of the box.
I know people like wallboards and monitors but we found them anti-pattern. If you find yourself looking at a wallboard/dashboard, it should already be an automated alert.
We’d receive an automatic email about a large circuit going down, we’d also receive a ticket about it; sometimes people dont look at the tickets closely enough, other times people get distracted with other topics, issues, etc. Having a large screen with interface status monitoring has proven to be effective enough; for example, someone walks by the monitor and says “why is this thing red, is it supposed to be?... and we immediately know one of the larger interfaces is down.
In an ideal world, we would not need it because every ticket will be diligently dealt with.... however in a real world, having a big red part of the screen flashing had proved quite effective.
I know people like wallboards and monitors but we found them anti-pattern. If you find yourself looking at a wallboard/dashboard, it should already be an automated alert.
Not at all. Alerts serve a different purpose. One of the most important things a team needs over the long haul is a feel for their system. Many people refer to this as mechanical sympathy. And the way you develop that is long-term exposure to rich data. Alerts are the red and yellow lights on your dashboard. But you get mechanical sympathy by listening to the sound of the engine, feel of the road, and the smell of th…
Lots of things have to be right for this to work, unfortunately, and company dashboards I've seen so far tend to be nowhere near it.
For instance, the dashboard refreshed $PERIOD only makes sense if you're showing data that updates $PERIOD, and if you can respond to changes in that data $PERIOD. $PERIOD = "in realtime" or "every minute" or "hourly" or whatever is relevant in a given context.
If you're looking at the dashboard much more frequently than the data changes, you're wasting time. If the data changes much more frequently than you're looking at it, you're likely to miss things, as 'geofft mentions elsewhere in the thread. And if you can't react to the data roughly as fast as it's updating, there's no point in looking at it so often. All those periods - recording, observing and reacting - must be roughly similar for the always-on dashboard to be useful, relative to generating reports every now and then.
Panels full of lights and charts work on fighter jets or on the bridge of the Enterprise, because the pilots/crew are in a tight feedback control loop with their dashboards.
(WRT. reacting in time, there are also error bars to consider. For instance, people on a diet are advised to weigh themselves weekly and not daily, because body mass varies by +/- 2kg during the day, so a naïve person checking weight daily would get fixated on those random oscillations. It's easier to tell regular people to reduce measurement frequency than to explain to them what a low-pass filter is and how is it relevant here. I have a feeling there's plenty of dashboard misuse that amounts to that too.)
--
Speaking of the Enterprise and "getting the feel for the system", there's something that I'd like to try one day: make a monitoring tool that translates various system metrics into background sounds, creating an ambience similar to the one you hear on the Enterprise-D[0][1]. I feel a somewhat unobtrusive mix of background noises would be better to develop "the feel for the system" than a visual dashboard. Real-life examples of this are combustion engine's RPM, or spinning rust hard drives, if anyone still remembers those.
--
[0] - https://www.youtube.com/watch?v=UKBvaOLDem0 - the bridge
[1] - In Enterprise's engineering, there's a well-known pulsating sound of the warp core; I can't find a good enough YouTube video (whatever there is, apparently got broken by YT's audio compression). This background pulsing correlated to the speed Enterprise was traveling with.
I know people like wallboards and monitors but we found them anti-pattern. If you find yourself looking at a wallboard/dashboard, it should already be an automated alert.
I’ll chime in here to say we use both at work. In a NOC at a medium-sized ISP, we are getting hammered with alerts 24/7. Some are not urgent, while others need to be actioned much faster - I mean 100G transit link down is no good. We’d receive an automatic email about a large circuit going down, we’d also receive a ticket about it; sometimes people dont look at the tickets closely enough, other times people get distr…
Alerts shouldn't be sent about anything that doesn't require an action.
Care to explain why one would use this over something much more capable like Grafana? [1] [1] https://github.com/grafana/grafana
For my use case ("check when the cronjob X on this machine last ran successfully"), setting up a data ingress pipeline which I could later configure as a time series data source seems like 3 times the effort it should actually take.
Earlier quoted context omitted.
> Why do you want a display of open PRs at all? All PRs are WIP, and minimizing WIP is very valuable in product development processes. See Reinertsen's The Principles of Product Development Flow for the math, but basically high/unpredictable latency drastically limits the pace of learning and causes a lot of upstream thrash and waste. I remember talking with one team at the bird-themed social media company that was f…
Yes, I understand why you'd want to focus on solving the number of open PRs. I agree that keeping that number down is good. My question is why do you want to put this on a TV screen . If you want people to focus on open PRs, tell them to open GitHub on their computers, don't tell them to look up at a TV screen periodically. Treat it like alerts: you have a list of open things to deal with and you need to get that num…
For example, consider the Kanban board. Here's one I built a while back: http://williampietri.com/writing/2015/the-big-board/
We loved having a physical map of what we were up to. We'd have our daily stand-up around board and discuss it. You'd know when something was completed, because you'd see somebody move a card. I would often know when the product manager was thinking about something he'd go over to look right at it. That often sparked conversations. And we'd all have a feel for how work was flowing, something we'd talk about in our weekly retro.
Could this have been replicated with a system of alerts? No. Alerts are interruptive and necessarily threshhold-driven. I don't want my people caught in a cycle of continuous reactivity to things that at some point in history were seen as important enough to configure an alert. Except for emergencies, I want them to be serene, thoughtful, and proactive, which is very hard to achieve if you're continuously juggling alerts.
So I'd put up something with PR stats if it were something I wanted us to be aware of. Especially so if it were an item of concern in previous retros. Maybe that would eventually lead to an alert (although I'd hope not). But the first step in solving a problem is understanding the problem, and I think information radiators are great for that, especially when problems are thorny and don't have obviously correct answers.
Earlier quoted context omitted.
By that logic a speedometer is an anti pattern and your car should just send up an alert when you're speeding... since when is getting accurate real-time information a bad thing?
That’s... actually true. It doesn’t matter how fast you’re going unless you’re speeding. And it distracts you by making you look down. The only reason we don’t have that yet is because the car doesn’t know the speed limit everywhere all the time.
* when speed restricted by equipment (trailer, temporary spare, etc)
* when observing advisory speeds
* when observing minimum speed requirements
* as a reference for judging appropriate speeds under inclement conditions
* as a reference for judging appropriate acceleration/deceleration rates when entering/exiting the roadway
Earlier quoted context omitted.
Because it cost like $10 and 5 minutes, could be set up by non-web-plumbers, and was pretty out of the box.
Aside: this is one of the biggest lessons of my adult life. Just because I could make something doesn't mean I should make something. Learning to value your time is a very underdeveloped skill.
Create a config file, and you get something similar.
Earlier quoted context omitted.
> I can’t tell you how many times I caught an issue because I knew our metrics backwards and forwards, but it didn’t trip an alert threshold. So how many times was an issue missed because you weren't in the office, or because you were looking at your own screen and not dashboards at the moment? Humans are incredibly powerful, but our whole job as SREs is to make things reliable, repeatable, and scalable. We're doing…
So how many times was an issue missed because you weren't in the office, or because you were looking at your own screen and not dashboards at the moment? That's not a problem with dashboards. That's a problem with training and staffing people. because you can't scalably fix problems with SSH. The number of businesses that need to worry about scalability is vanishingly small compared to the number of businesses that d…
Training and staffing people to look at dashboards? I've never heard of this and it sounds awful.