Teaching a new way to prevent outages at Google • Hacker News Board

Re: Teaching a new way to prevent outages at Google

#41

Post by hinkley » Fri, Mar 21, 2025, 6:52 AM UTC

nthingtohide wrote: post #40

decimalenough wrote: post #19

Earlier quoted context omitted.

If you're referring to the time they nuked an Australian retirement fund's VMware setup, no, that was basically a billing screwup. An operator left a field blank, the system assumed that meant a 1-year expiry, and dutifully deleted it after 1 year was up. https://cloud.google.com/blog/products/infrastructure/detail...

All mega deletes should be authorised. A human person should type in the word "delete" and then only the action should take place. Not doing this is like the decision is taken by VOID created by complex interacting systems.

Honestly unless it’s RTBF, no deletion should happen at all as long as you meet your reserve capacity of freshly silvered disks. Every defunct account should probable go to cold storage first.

Re: Teaching a new way to prevent outages at Google

#42

Post by nthingtohide » Fri, Mar 21, 2025, 11:03 AM UTC

hinkley wrote: post #41

nthingtohide wrote: post #40

Earlier quoted context omitted.

All mega deletes should be authorised. A human person should type in the word "delete" and then only the action should take place. Not doing this is like the decision is taken by VOID created by complex interacting systems.

Honestly unless it’s RTBF, no deletion should happen at all as long as you meet your reserve capacity of freshly silvered disks. Every defunct account should probable go to cold storage first.

We have sensible reasons to suggest this in both the cases : simple and complex.

If GCP is composed of 10-30 services (hypothetically) then keeping 5-10 employees whose job is ensure mega deletes are safe is not too much of a cost.

If GCP is composed of 500 services, then it is all the more important to have humans in the loop so ensure correct behaviour so that complex interacting services don't take a wrong action.

Re: Teaching a new way to prevent outages at Google

#43

Post by [deleted] » Sun, Mar 23, 2025, 9:27 AM UTC

perching_aix wrote: post #33

cynicalsecurity wrote: post #27

Earlier quoted context omitted.

The most unbelievable thing about that case was that Google actually deleted data instead of keeping then forever and use for ads.

Username checks out.

[deleted]

Re: Teaching a new way to prevent outages at Google

#44

Post by SlightlyLeftPad » Sun, Mar 30, 2025, 4:42 PM UTC

bbkane wrote: post #21

SlightlyLeftPad wrote: post #12

Earlier quoted context omitted.

This has really always been the case with Google philosophy docs. They tend to be very abstract and academic. The biggest danger is taking everything at face value and structuring your work or organization the same exact way based solely on these documents. The reality is, the vast majority of companies are not Google and will never encounter Google’s problems. That’s not where the value is though.

Maybe less of a philosophy doc, but I found the Google SRE workbook to have plenty of helpful concrete examples

Of course, I’m not suggesting they don’t contain great examples. It’s just silly to apply the book wholesale to a company 1/100th the size of Google or even 1/10th the size of Google. It’ll almost never work verbatim. You must adapt it to the organization, resources, architecture you have and adjust the direction for where you need it to go.

Someone else made the point that the book itself is an idealistic view as a visionary document of what Google wants it to be but from someone sitting in the SRE role at Google, the role is probably not exactly as described.