81–90 of 164 posts
It seems likely that some of the comments here are reacting to the headline as opposed to reading the article - which is one of my favourite pieces of writing on the subject of observability and responsible maintenance of large scale systems. Lots of great quotable bits in here. Here's just one for people who didn't make it to the bottom: "There’s a lot of daylight between just throwing your code over the wall and wa…
What would be a more accurate and neutral title?
>Nobody invests in their “test in prod” tooling. Firstly, what is logging then? How is this not tooling to ensure things are running smoothly? `less +F` anyone? Secondly, if you're running an aws/azure/gcp based server you now have a ridiculous amount of tooling for production diag, analytics and tracing.
"Firstly, what is logging then?" "Secondly, if you're running an aws/azure/gcp based server you now have a ridiculous amount of tooling for production diag, analytics and tracing." This presume that you, the person expected to fix problem X actually have access to logs and the servers where problems are happening. I've been at multiple projects/companies where this simply Isn't Allowed(tm). "You wrote this code, you…
Anyway, a good thing on this front now is that (speaking from Azure exp.), you can dump logs directly to blob storage (s3, whatever google call their data storage). Then you only need that permission.
As for the non-UTC servers... -_- but if it's a problem, you can always append the tz info to the log date format.
Earlier quoted context omitted.
The whole point of putting engineers on call (in this context) is to encourage them to make good technology choices and to take some ownership for their products. If there was no counter-pressure with pager duty or other threats to peaceful existence, then most developers would just pick whatever technology they personally enjoy using the most and expect that someone else will fix their special shitpile for them at 3…
There's more going on and worth exploring if engineers are so unattached to outcomes they pick technology in a vacuum. Pager duty is a heavy stick - it's important not to avoid root cause analysis. So if your engineers are making bad choices - what's really going on? That assumes engineers are even empowered to make technology choices. At many companies they are not (whether by dint of organizational structure or the…
Why would there be a process that could be executed that wouldn't already be automated? If the ops guy is dealing with an issue, it's because all the known remediations have failed.
Containers already have auto restart on failed health checks. VMs have vmotion and HA for failed hardware. If the ops guy is up at 3am, dealing with a service you wrote, chances are high that you (or your team) should be involved for the quickest resolution
"Engineers should be on call for their own code." - Would you rather work someplace you are expected to be on call 24/7, or a company that doesn't require that? It isn't the norm, and it isn't competitive. It's just more "always on" culture in the workplace - and that's not healthy. A company should understand workers need real breaks - and being on call is not a real break.
It seems likely that some of the comments here are reacting to the headline as opposed to reading the article - which is one of my favourite pieces of writing on the subject of observability and responsible maintenance of large scale systems. Lots of great quotable bits in here. Here's just one for people who didn't make it to the bottom: "There’s a lot of daylight between just throwing your code over the wall and wa…
What would be a more accurate and neutral title?
Most places that I've worked, that "test in prod", do so for one reason. It all comes down to laziness. The first scenario is they have never set up a test environment in the first place. They're either too lazy to do so, or too lazy to look into how to do it. Often confused with being 'too busy to do it'. The second scenario is that they do have a testing environment-however, for some reason, it's broken. Some chang…
One lesson of modern architectures (i.e. anything more recent than the LiveJournal-style Web/App/DB 3-tier stack) is that it is literally impossible to create and maintain a test environment that has enough similarity to prod to be useful.
For the overwhelming majority of non-prod environments and release processes I've seen they usually have basic problems and would catch a lot more issues with additional engineering investment. I.e Releases aren't automated, self-contained, or idempotent. Unit testing coverage is bad, integration tests are non-existent, data used isn't reflective of reality, there is no performance or load testing done, downstream or upstream systems have poorly defined interfaces and are excessively mocked or just not considered.
Earlier quoted context omitted.
"Firstly, what is logging then?" "Secondly, if you're running an aws/azure/gcp based server you now have a ridiculous amount of tooling for production diag, analytics and tracing." This presume that you, the person expected to fix problem X actually have access to logs and the servers where problems are happening. I've been at multiple projects/companies where this simply Isn't Allowed(tm). "You wrote this code, you…
Feel your pain I know what it's like to ask someone in charge of 8k people to personally approve my server permission escalation even though he's never met me, doesn't know anything about the IT systems, has never visited the site I work at or probably even been in the country I reside. Anyway, a good thing on this front now is that (speaking from Azure exp.), you can dump logs directly to blob storage (s3, whatever…