111–120 of 840 posts
Duplicate of https://news.ycombinator.com/item?id=25632048 I think HN is hiding these posts. Maybe status threads are discouraged now? But they're much more useful than status.slack.com etc.
HN is where we all go when the Internet (or large portions of it) are down. It's more reliable than all the 'downforeveryoneorjustme' or 'downtime monitor' services.
Earlier quoted context omitted.
To be fair, IRC doesn't do a lot of things Slack does. Where is the logging and audit trails, access control, search, etc.
Several IRC servers do have support for authentication and access control (and audit trails as well I suppose). Only centralized history/logging and search would need to be bolted on if needed. In the non-centralized case your IRC client takes care of all of that.
When I was at Uber, we noticed that most incidents are directly caused by human actions that modify the state of the system. Therefore, a large "backlog" of human actions that modify the system state have a much higher chance of causing an incident. My bet is that this incident is caused by a big release after a post-holiday "code freeze".
I would bet it's just the influx of traffic post holiday with systems that haven't been updated in so long maybe some annoying memory leaks have crept up and gone unnoticed or some other bad state that was exacerbated by return to work day for most NA folks. Code freezes were good at identifying bugs that only show up after long periods. Doubt anyone releasing big changes Monday morning.
Earlier quoted context omitted.
I wouldn't be shocked if businesses saw increased productivity during these.
I'm sitting here not sure if I should deploy code since most communication with the rest of my team has been cut off.
Using our teams backup chatroom in a competing service. One of these days P2P Matrix will reach GA, then I plan to make a backup for my backups, Starfleet style.
That's one obscure reference, I love it. https://www.youtube.com/watch?v=UaPkSU8DNfY GILORA: Starfleet code requires a second backup? O'BRIEN: In case the first backup fails. GILORA: What are the chances that both a primary system and its backup would fail at the same time? O'BRIEN: It's very unlikely, but in a crunch I wouldn't like to be caught without a second backup.
Earlier quoted context omitted.
Do you guys not have e-mail? looks through Inbox of 850 new aws, batch job and logging messages oh yea, that's right..
Don't you guys have e-mail filters? "Hey, our site has been down for 2 hours, why aren't you doing anything" Looks at 850 unread messages in ops-notifications folder ooh yeah, that's right..
In my organization it's spelt "deleted items"
Earlier quoted context omitted.
EFnet was always splitting every few hours. I don't really miss IRC compared to modern chat systems.
I'm the opposite. Back when in my early teens, friends and I would attempt to hijack opposing groups' channels via takeovers during net-splits (and ofcourse having the same done to us). What a time to be alive.
Earlier quoted context omitted.
then wouldn't this happen every monday morning?
A lot of organizations essentially took the last two weeks off from work, which is long enough for a 10-day autoscale window to spin down servers, and then get confronted by a load spike that wasn't pre-spun for.