I have been following the changes on GitHub in the last 2 days, I wish they would actually add comments to the commits to explain the reasons of change or further info.
I think they have issues with message retrying and spamming the server.
- They added handling of HTTP 508 and do not retry jobs in this case
- There is a list of HTTP errors that are handled manually, including the new HTTP 508, but for example 503 is not handled and Jobs (e.g. messages) will be retried indefinitely and I have seen logs with many 503 entries and retries.
- They have increased the maximum backoff time from 30s to 60s (+ added jitter) and it can now also be changed remotely via server flag.
- Any HTTP error not handled explicitly is considered an IOException (strange?) and retried. List of handled HTTP exceptions (413, 401, 403, 409, 410, 411, 423, 499 and 508) is here: https://github.com/signalapp/Signal-Android/blob/2c1c6fab356... I personally would not repeat requests automatically by default on any HTTP error - unless handled manually (so reverse logic), only on connection error.
- They have just switched off OkHTTP automatic "silent" connection retries: https://square.github.io/okhttp/3.x/okhttp/okhttp3/OkHttpCli...
I don't think they have tested this enough, it's more of a blind try and I think it can decrease request delivery reliability.
The OkHttp implementation retries requests silently (with backoff and retry limit) in case of: Unreachable IP addresses, Stale pooled connections, Unreachable proxy servers. This is transparent to the application. In the past I have complained to the OkHttp team that POST requests should not be retried at all and they fixed it (your requests need to be idempotent anyway, regardless of this).
- I believe they may be also having issues with requests that are non idempotent and have been retried by OkHttp silently or in cases where application has sent something to the server and server received it, but the response has never made it back to the client. In this case client will retry request and this can cause trouble if requests are not deduplicated on server side and idempotent. But this is just my guess.