AWS Operational issue – Multiple services in us-east-1

Post by mbaumbach » Wed, Jul 31, 2024, 12:43 AM UTC

tcas wrote: post #9

My guess is this is all due to CloudWatch logs putlogevents failures. By default a docker container configured with awslogs runs in "blocking" mode. As logs get logged, docker will buffer them and push to CloudWatch logs frequently. In case the log stream is faster than what the buffer can absorb, stdout/stderr get blocked and then the container will freeze on the logging write call. If putlogevents is failing, buffe…

Post by tcas » Wed, Jul 31, 2024, 2:01 AM UTC

mbaumbach wrote: post #11

tcas wrote: post #9

My guess is this is all due to CloudWatch logs putlogevents failures. By default a docker container configured with awslogs runs in "blocking" mode. As logs get logged, docker will buffer them and push to CloudWatch logs frequently. In case the log stream is faster than what the buffer can absorb, stdout/stderr get blocked and then the container will freeze on the logging write call. If putlogevents is failing, buffe…

I just want to thank you for providing this info. This was exactly the cause of some of our issues and this config setting restored functionality to a major part of our app.

Post by remram » Wed, Jul 31, 2024, 2:56 AM UTC

ackdesha wrote: post #6

It seems to have cascaded from AWS Kinesis... [03:59 PM PDT] We can confirm increased error rates and latencies for Kinesis APIs within the US-EAST-1 Region. We have identified the root cause and are actively working to resolve the issue. As a result of this issue, other services, such as CloudWatch, are also experiencing increase error rates and delayed Cloudwatch log delivery. We will continue to keep you updated a…