How does a single network card emitting bad packets effect other sites? > investigations into the logs, including packet captures, was occurring in tandem, which ultimately identified a suspected card issue in Denver, CO. Field Operations were dispatched to remove the card. Once removed, it did not appear there had been significant improvement; however, the logs were further scrutinized .. to identify that the source…
lookupByCity(...) {
....
try {
conn = connectionPool.getConnection();
stmt = conn.createStatement();
...
} finally {
if (stmt != null) {
stmt.close();
}
if (conn != null) {
conn.close();
}
}
}
close() can throw, and in the circumstance of the outage it did for stmt, leading to the connection not getting closed and eventually the pool being exhausted with every thread blocked waiting for a connection. It's an interesting chain of failures, arguably the presence of such a chain is the real root cause, rather than the unhandled sql exception.