41–50 of 283 posts
This is a fun debugging story, but is a great example why servers should be cattle not pets. Having trouble with a VM? Blow it up and get a fresh one. Still having trouble? The provisioning steps are codified, you can walk through them and find the one that causes the issue.
that’s a pretty good and detailed explanation. 2 things: 1) i hope they have a runbook for situations like this (ie the support engineer does not have to figure all this on the fly) 2) the customer should have provides more details and maybe should have thought of the tweaks they made (classic solution is to compare 2 instances - one works one does not)
1) you cannot have a runbook for everything, and even if you have a runbook you the best you could have found in this case is that something weird was happening in the VM. The setting had an insanely big value but it was accepted by the kernel, so you would assume it was a valid one. 2) the customer provided a huge amount of details, but it is usually very hard to explain what did you change from the base image. Most…
This is a fun debugging story, but is a great example why servers should be cattle not pets. Having trouble with a VM? Blow it up and get a fresh one. Still having trouble? The provisioning steps are codified, you can walk through them and find the one that causes the issue.
That's awfully convenient, and I can't deny having done this, but it's also a great way to never understand what went wrong.
> After another spin around the world the case comes back to our team. So basically, follow the sun doesn't work for hard problems? Can you really say the people are working on this 24/7 if progress is only made in one time zone?
This is a fun debugging story, but is a great example why servers should be cattle not pets. Having trouble with a VM? Blow it up and get a fresh one. Still having trouble? The provisioning steps are codified, you can walk through them and find the one that causes the issue.
If a freshly provisioned VM doesn't have the same issue, then they're not using automated configuration management (Puppet, Chef, or similar) for these settings, and then they have a more serious problem, as nothing in their runtime environment is "codified" or predictable.
if (rmem > (size + sk->sk_rcvbuf)) goto uncharge_drop; What is rmem in this case? I'm a bit confused as to why it is written that way. This drops the packet right when it overflows the buffer?
> they use raw sockets! Raw sockets are different than normal sockets: they bypass iptables But this bugreport says raw sockets would be filtered by the OUTPUT chain of iptables: https://bugzilla.redhat.com/show_bug.cgi?id=1269914#c4 Is that accurate across distros? It does make sense for some socket types, like device sockets, to not be routed through iptables.
Earlier quoted context omitted.
1) you cannot have a runbook for everything, and even if you have a runbook you the best you could have found in this case is that something weird was happening in the VM. The setting had an insanely big value but it was accepted by the kernel, so you would assume it was a valid one. 2) the customer provided a huge amount of details, but it is usually very hard to explain what did you change from the base image. Most…
i’m not saying a runbook will catch everything. but it will give you a chance to solve the problem quicker.