Only 100 bytes? That's easy... Sheesh.
[deleted]
Lessons from a Google App Engine SRE on how to serve over 100B requests per day
21–30 of 30 posts
Re: Lessons from a Google App Engine SRE on how to serve over 100B requests per day
#22Earlier quoted context omitted.
SREs are very hard to hire, speaking from experience. At Google SRE directors and VPs will often cherry-pick promising candidates from the mainline SWE hiring pipeline and give them a "hero call" to convert them to SREs. SREs at Google are also paid more, controlling for level and performance, as a way to hire and retain.
Interesting. Can you expand on "hero call"? What does that entail?
Donning a cape and meeting destiny.
In all seriousness they make it out to be more than it is. From my experience going through their hiring pipeline there seem to be two tracks in SRE; software and sysadmin. If you score higher in algorithms and data-structures, presumably, you'll end up working more on tools and libraries whereas in the other you'll work more on infrastructure and automation. Either way both tracks work together on the same team towards the same goals.
If you want in be prepared to solve simple-to-tough algorithms problems and be quizzed on TCP re-transmission, Linux system calls, and memory pressure. It's a bit challenging because you not only have to know Big-O well enough to estimate the asymptotic complexity of an arbitrary algorithm but you might also be asked what a sequence of TCP packets would look like if you sent some data and pulled the plug or what the parameters are to a given system call on Linux. You quite literally have to know everything from how virtual memory works, how to implement a fast k-means, how the network stack works from top to bottom, etc, etc.
If you've done any work in cloud development and supporting moderately large one it's that but bigger. Make one a hero, it does not.
Re: Lessons from a Google App Engine SRE on how to serve over 100B requests per day
#23Earlier quoted context omitted.
SREs are very hard to hire, speaking from experience. At Google SRE directors and VPs will often cherry-pick promising candidates from the mainline SWE hiring pipeline and give them a "hero call" to convert them to SREs. SREs at Google are also paid more, controlling for level and performance, as a way to hire and retain.
Interesting. Can you expand on "hero call"? What does that entail?
It's just a guy on the phone telling you how you won't be like those other chumps, you'll be a hero. The few and the proud. Standard recruitment techniques.
Re: Lessons from a Google App Engine SRE on how to serve over 100B requests per day
#24The book they mention[1] is very good so far. [1] https://play.google.com/store/books/details/Betsy_Beyer_Site...
Upvoting because this link is cheaper than the Amazon link.
If you are willing to wait, many times of the year O'Reilly will have their ebooks on sale for 50%-60% off. The best time is black friday.
Re: Lessons from a Google App Engine SRE on how to serve over 100B requests per day
#25I'm actually really really glad that Google released this book because I think they are one of the few companies that is actually doing this SRE thing right. I think the hardest bit about the SRE paradigm (like DevOps) is having companies wholly adopt it, and I think that this book being out will help change that.
Re: Lessons from a Google App Engine SRE on how to serve over 100B requests per day
#26I've been seeing a lot of references to SRE recently. Is Google trying to market this position and acquire more engineers? The SRE book, and Google in general, have mentioned that SREs are notoriously hard to hire, and I'm wondering if they are doing a marketing push.
> I've been seeing a lot of references to SRE recently. Is Google trying to market this position and acquire more engineers?
There is a bit of a gap (in terms of attitude and skill set) between what Google calls an SRE and what most other companies call an SRE.
I think Google is trying to steer the public usage of the word so that their term doesn't get diluted. One possible reason might be so that SREs at Google don't feel like they're making a bad career move by having the term "SRE" on their resume.
If you spent 5 years working for the state of California, designing safer future-proof and state-of-the-art treatment plans and plants for gray and black waste water in metropolitan and rural areas, improving life expectancy by 2% for people who live in California, but your title was "Sanitation Engineer" the whole time, you're going to be a bit put out if you learn that during that time all the high schools in the state changed the custodians' titles to the same thing.
Re: Lessons from a Google App Engine SRE on how to serve over 100B requests per day
#27This got me wondering what the AWS services' work load per day was. Best numbers I could find were from this 2013 article about serving ≈95 billion requests per day for just S3. The size and scope of cloud providers is truly cool and fascinating engineering.
https://aws.amazon.com/blogs/aws/amazon-s3-two-trillion-obje...
Re: Lessons from a Google App Engine SRE on how to serve over 100B requests per day
#28> Advance preparation, combined with extensive testing and contingency plans, meant that we were ready when things went [slightly wrong] and were able to minimize the impact on customers.
Following that link provided some interesting reading (for a mundane error report, at least): https://groups.google.com/forum/#!msg/google-appengine-downt...
TIL that even Google have datacenter fluctuations they can't figure out. It's nice that they quietly make this info publicly available, and also nice that I've now discovered where to find it :)
Re: Lessons from a Google App Engine SRE on how to serve over 100B requests per day
#29s/lessons/lesson/
"If you put a human on a process that’s boring and repetitive, you’ll notice errors creeping up. Computers’ response times to failures are also much faster than ours. In the time it takes us to notice the error the computer has already moved the traffic to another data center, keeping the service up and running. It’s better to have people do things people are good at and computers do things computers are good at."
Re: Lessons from a Google App Engine SRE on how to serve over 100B requests per day
#30> Advance preparation, combined with extensive testing and contingency plans, meant that we were ready when things went [slightly wrong] and were able to minimize the impact on customers. Following that link provided some interesting reading (for a mundane error report, at least): https://groups.google.com/forum/#!msg/google-appengine-downt... TIL that even Google have datacenter fluctuations they can't figure out. I…
I love their solution.
They turned it off, then on again