Monitoring Microservices with Synthetic Transactions in Go

- constant strings (defined as vars) as return values instead of the standard error type

- not formatted

- synthetic request's response body not closed - a memory leak

- log.Fatal (which results in os.Exit(1) immediately killing the program) for all the non-critical errors

- timestamp as a string instead of time.Time in the struct

- not re-using http.DefaultClient

- 3rd party package with a custom DSL instead of a simple time.Ticker

Sorry for the pedantry, but if such code examples live on the internet, newcomers to Go will learn from it (if not blindly copy-paste) and it will spread even further indirectly giving bad reputation to the language.

P.S. i do like this idea of continuous integration testing in production, will probably start using it at our company.

The microservices trend is great but this sort of monitoring starts to fall apart when you horizontally scale for HA. In this case we separate out our monitoring by doing synthetic requests at the service level, and then very specific health checking at the "instance" level (really containers now). Instance level health checks will ensure connectivity to outside dependencies, databases, filesystems etc. The trick is to know when a failure is a localized failure or a widespread failure. No sense in taking out all of the instances if they all can't talk to the DB.

Also, I'm not a Go developer but is this idiomatic?

The error handling is kinda strange.

I would do some like this:

    var ErrUnexpectedResponse = errors.New("unexpected response")
    ....
    func syntheticHttpRequest(url string, apiToken string) error {
        ...
        } else if resp.StatusCode != 202 {
            return ErrUnexpectedResponse
        }
        return nil
    }

We have a large, complex pipeline for processing incoming data. Historically it has been very opaque which made debugging data errors hard.

We added event logging throughout the pipeline. We are building a tool that feeds known data in one end and checks the final output. We can then use the event logging in between to monitor state throughout for errors. It also allows us to see how long it takes to transition between events.

Ideally the tool will check every event for deviations from the expected output and alert us if any events fail. It will also alert us if the time between events rises above the average by a predefined threshold.

I'd love to hear about any open source tools designed to do something like this.