Nobody likes lag: How to make low-latency dev sandboxes
1–10 of 47 posts
Re: Nobody likes lag: How to make low-latency dev sandboxes
#2Not directly related but can't read the text on my phone. It's too thin, maybe you could increase the font weight a bit?
Re: Nobody likes lag: How to make low-latency dev sandboxes
#3Great write-up on the evolution of your architecture. The progression from 200ms → 14ms is impressive.
The lesson about "delete code to improve performance" resonates. I've been down similar paths where adding middleware/routing layers seemed like good abstractions, but they ended up being the performance bottleneck.
A few thoughts on this approach:
1. Warm pools are brilliant but expensive - how are you handling the economics? With multi-region pools, you're essentially paying for idle capacity across multiple data centers. I'm curious how you balance pool size vs. cold start probability.
2. Fly's replay mechanism is clever, but that initial bounce still adds latency. Have you considered using GeoDNS to route users to the correct regional endpoint from the start? Though I imagine the caching makes this a non-issue after the first request.
3. For the JWT approach - are you rotating these tokens per-session? Just thinking about the security implications if someone intercepts the token.
The 79ms → 14ms improvement is night and day for developer experience. Latency under 20ms feels instant to humans, so you've hit that sweet spot.
Re: Nobody likes lag: How to make low-latency dev sandboxes
#4I’m experiencing a similar issue hosting MCP Server on Cloud Run with scale-to-zero for cost optimization. As far as I know, Cloud Functions v2 and Cloud Run both are container-based, and they tend to have noticeable startup times.
In contrast, AWS Lambdas, which run on Firecracker, have sub-second startup latency, often just a few hundred milliseconds.
Is there anything comparable on GCP that achieves similar low latency cold starts?
Re: Nobody likes lag: How to make low-latency dev sandboxes
#5@mnazzaro have you seen fly.io's new sprites.dev offering?
Re: Nobody likes lag: How to make low-latency dev sandboxes
#6I’m experiencing a similar issue hosting MCP Server on Cloud Run with scale-to-zero for cost optimization. As far as I know, Cloud Functions v2 and Cloud Run both are container-based, and they tend to have noticeable startup times. In contrast, AWS Lambdas, which run on Firecracker, have sub-second startup latency, often just a few hundred milliseconds. Is there anything comparable on GCP that achieves similar low la…
I'm a huge GCP fan, but cloud run wouldn't fit our use case because of the routing and ephemeral nature. I think you would have to try to build something yourself using GKE + gVisor
Re: Nobody likes lag: How to make low-latency dev sandboxes
#7Great write-up on the evolution of your architecture. The progression from 200ms → 14ms is impressive. The lesson about "delete code to improve performance" resonates. I've been down similar paths where adding middleware/routing layers seemed like good abstractions, but they ended up being the performance bottleneck. A few thoughts on this approach: 1. Warm pools are brilliant but expensive - how are you handling the…
1. The pools are very shallow- two machines per pool. While it's certainly possible for 3 tasks to get requested in the same region within 30 seconds, we handle that by falling back to the next closest region if a pool is empty. This is uncommon, though.
2. I haven't considered it, but yeah- the caching seems to work great for us.
3. The tokens are generated per-task, so if you are worried about your token getting leaked, you can just delete the task!
Re: Nobody likes lag: How to make low-latency dev sandboxes
#8@mnazzaro have you seen fly.io's new sprites.dev offering?
I have! It's pretty interesting and handles a lot of the problems discussed here, but is a little young for us. For one thing, it doesn't have fly replay, so we'd have to build a separate proxy again.
If we were starting from 0, I would definitely try it. My favorite thing about it is the progressive checkpointing- you can snapshot file system deltas and store them at s3 prices. Cool stuff!
Re: Nobody likes lag: How to make low-latency dev sandboxes
#9Not directly related but can't read the text on my phone. It's too thin, maybe you could increase the font weight a bit?
Thanks for letting me know- I'll take a look
Re: Nobody likes lag: How to make low-latency dev sandboxes
#10When Covid hit I wasn’t the only one working remotely at my company, but I was the only one working remotely in North America, and apparently the only one trying to Work Smarter. By then there were a handful of feature toggles I had implemented that I quickly set to always on in development, but chief among them was that gzip service calls were a net loss in AWS but very very handy while working from home.
I also had switched a head of line service call that was, for reasons I never sorted out, costing us 30ms TTFB per request for basically fifty bytes of data, to use a long poll in Consul because the data was only meant to be changed at most once every half hour and in practice twice a week. So that latency was hidden in dev sandbox except for startup time, where we had several consul keys being fetched in parallel and applied in order, so one more was hardly noticeable.
The nasty one though was that Artifactory didn’t compress its REST responses, and when you have a CI/CD pipeline that’s been running for six years with half a hundred devs that response is huge because npm is teh dumb. So our poor UI lead kept having npm install timeout and the UI team’s answer for “my environment isn’t working” started with clearing your downloaded deps and starting over.
They finally fixed it after we (and presumably half of the rest of their customers) complained but I was on the back 9 of migrating our entire deployment pipeline to docker and so I had nginx config fairly fresh in my brain and I set them up a forward proxy to do compression termination. It still blew up once a week but that was better than him spending half his day praying to the gods of chaos.