31–40 of 128 posts
If you're using AWS resources, give LocalStack a try for integration testing
Maybe things have improved since couple years ago, though
Recently, I've heard of several companies folding up SRE and moving individuals to their SWE teams. Rumors are that LinkedIn, Adobe, and Robinhood have done this. This made me think: is SRE a byproduct of a bubble economy of easy money? Why not operate without the significant added expense of SRE teams? I wonder if SRE will be around 10 years from now, much like Sys Admins and QA testers have mostly disappeared. Inst…
I'm a SWE SRE. I think in some cases it is better to be folded into a team. In other cases, less so.
One SRE team can support many different dev teams, and often the dev teams are not at all focusing time on the very complicated infra/distributed systems aspect of their job, it's just not something that they worry about day to day.
So it makes sense to have an 'infra' that operates at a different granularity than specialized dev teams.
That may or may not need to be called SRE, or maybe it's an SRE SWE team, or maybe you just call it 'infrastructure' but at a certain scale you have more cross cutting concerns across teams where it's cheaper to split things out that way.
https://x.com/alexpotato/status/1432302823383998471?s=20
Teaser: "25. If you have a rules engine where it's easier to make a new rule than to find an existing rule based on filter criteria: you will end up with lots of duplicate rules."
Something I hope to eventually hear is the solution to the full cold start problem. Most giant custom-stack companies have circular dependencies on core infrastructure. Software-defined networking needs some software running to start routing packets again, diskless machines need some storage to boot from, authentication services need access to storage to start handing out service credentials to bootstrap secure authz…
IIRC some of the information about their approach is considered sensitive so I won't elaborate further.
This is a great writeup, and very broadly applicable. I don't see any "this would only apply at Google" in here. > COMMUNICATION CHANNELS! AND BACKUP CHANNELS!! AND BACKUPS FOR THOSE BACKUP CHANNELS!!! Yes! At Netflix, when we picked vendors for systems that we used during an outage, we always had to make sure they were not on AWS. At reddit we had a server in the office with a backup IRC server in case the main one…
Afair Google just ran irc on their corp network which was completely separate from prod so I wouldn’t be surprised if it was in a small server room in the office somewhere. > very broadly applicable. I don't see any "this would only apply at Google" in here. One thing I haven’t even heard of anyone else doing was production panic rooms - a secure room with backup vpn to prod
I thought Google didn't have a "corp" network because of their embrace of zero-trust in BeyondCorp?
Recently, I've heard of several companies folding up SRE and moving individuals to their SWE teams. Rumors are that LinkedIn, Adobe, and Robinhood have done this. This made me think: is SRE a byproduct of a bubble economy of easy money? Why not operate without the significant added expense of SRE teams? I wonder if SRE will be around 10 years from now, much like Sys Admins and QA testers have mostly disappeared. Inst…
Still the same competency. Someone needs to know how those protocols work, tell you latency characteristics of storage, and read those core dumps.
Off topic: TIL Google has its own TLD (.google)
I'm curious how people approach big red buttons and intentional graceful degradation in practice, and especially how to ensure that these work when the system is experiencing problems. E.g. do you use db-based "feature flags"? What do you do then if the DB itself is overloaded, or the API through which you access the DB? Or do you use static startup flags (e.g. env variables)? How do you ensure these get rolled out q…
Earlier quoted context omitted.
Afair Google just ran irc on their corp network which was completely separate from prod so I wouldn’t be surprised if it was in a small server room in the office somewhere. > very broadly applicable. I don't see any "this would only apply at Google" in here. One thing I haven’t even heard of anyone else doing was production panic rooms - a secure room with backup vpn to prod
> Afair Google just ran irc on their corp network which was completely separate from prod I thought Google didn't have a "corp" network because of their embrace of zero-trust in BeyondCorp?