I had a GCP Cloud Run function that rendered videos. It was fine for one video per request but after that it slowed to a crawl and needed to shut down to clear out whatever was wrong. I assume a memory leak in MoviePy? Spent a couple of days looking at multiple options and trying different things, in the end I just duplicated the service so I had three of them and rotated which one we sent video renders to and do eac…
It worked great and was way easier to deploy than k8s setups. However after some testing we found out that the core logic of the app - a long running process, would just crawl to a halt after some time.
It turned out google wanted to push you to use their (paid) queue / pubsub solutions, but they didn’t want it to be _too_ obvious, so cloudrun would actually throttle its cpu sometime after the request it was spawned was returned.
Our logic was based on pushing stuff in a queue and having it be processed outside of a request, but google just f*ked with that solution.
And it would have been fine if that was upfront info, but it was buried in a doc page somewhere obscure, small print…
Thats the time I realized how bad gcp can be…