Earlier quoted context omitted.
Hey, I am one of the authors of the article and the systems described. The default 20% margin of error is indeed pretty wide, and it is intended to catch large obvious regressions (e.g. an algorithm accidentally becoming quadratic instead of being linear) As we described in the blog post, we have the second system based on the real hardware. This system is on-demand. If an engineer has a suspect commit or an experime…
Thanks for your reply. Can you share how these laptops are managed? I took another look at the article but didn't find that information there. I.e I'm curious if there's a cloud provider managing them for you or you guys keep them in a closet somewhere.
For things like system updates and taking care of the hardware - we do it manually today. The fleet is still small, so it is manageable but in the future we would like to consider a vendor, if we can find one.