>Now we run automated tests to monitor server uptime and check server for problems every 30 seconds. Unfortunately automated test scripts were happily getting HTTP/200 replies while people using the Chrome browser were being told this is a scam business trying to steal their bank account information. I was surprised this wasn't part of the lessons learned. But it seems the monitoring basically failed but that wasn't…
When talking about checks on the order of twice a minute, curl is probably the right approach. You can/should still do a full check, but that can be done at a lower frequency.
For example:
- a frequent/simple check dealing directly (on the internal network) with the webserver ("does it work well yes/no, what's the raw response time, etc..."). Here is where I would definitely use "curl".
- another less frequent test involving as well the DNS and the external network.
- another end-to-end test (e.g. once every 10 minutes?) involving as well one or more real browsers (this would test as well for example revoked SSL certs).
=> all these infos/metrics should be quite helpful to identify problems, or at least to shrink the potential area that is causing it.