All of us test in production all the time (2019) • Hacker News Board

Re: All of us test in production all the time (2019)

#131

Post by ritchiea » Mon, Jul 20, 2020, 5:31 AM UTC

anon9001 wrote: post #120

faizshah wrote: post #118

Earlier quoted context omitted.

Wholeheartedly agree to everything you’ve said. I have worked at a lot of places where automated testing is almost non existent. Instead they use staging/testing environments and manual testing. For all these projects they frequently have downtime or errors in prod where something was uncaught. It takes ages to release a fix because they have to carefully research how a change affects other code instead of running a…

> They are convinced that unit tests are unnecessary extra work for their project but they don’t realize they are losing time by having to manually test and research changes. If people are finding tests unnecessary extra work, maybe they're right for their project. Most jobs are just "ship feature asap, try not to break anything, repeat". Most managers don't care about code quality as much as speed of delivery. Most…

Managers need developers to write tests so that when that developer job hops the new dev on the team can actually make changes with confidence. As it happens, writing tests and adhering to best practices has a side effect of keeping turnover low because developers can do their jobs with less friction.

Re: All of us test in production all the time (2019)

#132

Post by cs02rm0 » Mon, Jul 20, 2020, 5:35 AM UTC

anon9001 wrote: post #120

faizshah wrote: post #118

Earlier quoted context omitted.

Wholeheartedly agree to everything you’ve said. I have worked at a lot of places where automated testing is almost non existent. Instead they use staging/testing environments and manual testing. For all these projects they frequently have downtime or errors in prod where something was uncaught. It takes ages to release a fix because they have to carefully research how a change affects other code instead of running a…

> They are convinced that unit tests are unnecessary extra work for their project but they don’t realize they are losing time by having to manually test and research changes. If people are finding tests unnecessary extra work, maybe they're right for their project. Most jobs are just "ship feature asap, try not to break anything, repeat". Most managers don't care about code quality as much as speed of delivery. Most…

I don't disagree, probably.

I keep finding myself working on internal projects that are driven by a UI and have little business logic but lots of I/O to other systems. With no hard requirements, no guarantee of shipping and no direct access to end users. Which I find less suited to unit tests than (automated) integration testing. As soon as you put unit testing in all anyone can ask is what your test coverage percentage is and why it's some value less than 100.

Unit testing is a dev tool, if a dev sees value in writing and maintaining a given test, great, otherwise it could just be that it's not a panacea. Let them strike a balance but ensure the team ask the question of any production issues - could a unit test have prevented this?

Re: All of us test in production all the time (2019)

#133

Post by t-writescode » Mon, Jul 20, 2020, 5:36 AM UTC

Would a better word be "validating" in production?

I try to validate all of my changes in production, see them run, see them output expected values, etc.

I try very hard to test everything I can prior to production; but, then I read loglines, watch graphs or run the code myself, all to validate that all the interconnected pieces are working together.

Re: All of us test in production all the time (2019)

#134

Post by t-writescode » Mon, Jul 20, 2020, 5:42 AM UTC

danpozmanter wrote: post #42

"Engineers should be on call for their own code." - Would you rather work someplace you are expected to be on call 24/7, or a company that doesn't require that? It isn't the norm, and it isn't competitive. It's just more "always on" culture in the workplace - and that's not healthy. A company should understand workers need real breaks - and being on call is not a real break.

I think the model that I prefer is: "if you're going to deploy some code in the evenings or beginning of the weekend, you're on call for the code you added until work day tomorrow"

And have it be only around the code you just added.

If you deploy (and validate) earlier in the day or not on Friday ( or whatever the end-of-week day is) then that requirement is gone.

I've definitely seen code get deployed at 8pm on a Friday night; and, who knows if it was guaranteed to work. That person definitely should be on call for it.

Re: All of us test in production all the time (2019)

#135

Post by ratww » Mon, Jul 20, 2020, 8:18 AM UTC

tluyben2 wrote: post #102

ratww wrote: post #75

Earlier quoted context omitted.

People don't say it directly, but all the constant complaining about the bloat and complexity of the JS ecosystem and the size of node_modules folder boils down mostly to two tools: Babel and Webpack.

But that is a sign is it not? If tools as badly designed as webpack is the standard, is that not saying there is an issue with the entire ecosystem?

There are other standard tools that don't suffer with package bloat, like Prettier and Typescript, which don't have dependencies. This issue is not inherent to the ecosystem.

Re: All of us test in production all the time (2019)

#136

Post by devdas » Mon, Jul 20, 2020, 8:23 AM UTC

ratww wrote: post #67

_diq5 wrote: post #55

Earlier quoted context omitted.

Want to jump in here - I have worked at a company where engineers are not on call for their code, and it was a living nightmare. _You_ might not be on call for your code, but _somebody_ will be. Often some poor SRE/ops person that has absolutely no idea what the app is doing/or why it's failing in production. Not being on-call makes engineers complicit. I've seen it all, known memory leaks shipped into production, ap…

I'm always happy to help some poor SRE in the middle of the night, and I once even drove to the office in a rainy Sunday, in the middle of my vacation, to access IP-restricted stuff because a support intern messaged me on Instagram. ...but with that said: I'm glad I only worked in countries where work is properly regulated and "on call" means "I'm getting fucking paid every cent for each hour I _must_ answer that god…

Now apply the same rules to the SRE role as well.

Re: All of us test in production all the time (2019)

#137

Post by devdas » Mon, Jul 20, 2020, 8:26 AM UTC

danpozmanter wrote: post #68

bob1029 wrote: post #64

Earlier quoted context omitted.

The whole point of putting engineers on call (in this context) is to encourage them to make good technology choices and to take some ownership for their products. If there was no counter-pressure with pager duty or other threats to peaceful existence, then most developers would just pick whatever technology they personally enjoy using the most and expect that someone else will fix their special shitpile for them at 3…

There's more going on and worth exploring if engineers are so unattached to outcomes they pick technology in a vacuum. Pager duty is a heavy stick - it's important not to avoid root cause analysis. So if your engineers are making bad choices - what's really going on? That assumes engineers are even empowered to make technology choices. At many companies they are not (whether by dint of organizational structure or the…

A documented process can be automated.

Humans should be paged only when this is a new category of failure, and in that case, having the developer wake up first triggers a really good feedback loop.

Re: All of us test in production all the time (2019)

#138

Post by ratww » Mon, Jul 20, 2020, 8:29 AM UTC

ZephyrBlu wrote: post #105

ratww wrote: post #75

Earlier quoted context omitted.

People don't say it directly, but all the constant complaining about the bloat and complexity of the JS ecosystem and the size of node_modules folder boils down mostly to two tools: Babel and Webpack.

I don't think it's the tools' fault. They were built to solve problems, and they solve those problems quite well IMO.

Sure, but I don't think those things are related. It's possible to solve problems without causing package bloat. Typescript and Prettier have no dependencies, for instance.

Re: All of us test in production all the time (2019)

#139

Post by truculent » Mon, Jul 20, 2020, 9:20 AM UTC

aunty_helen wrote: post #32

>Nobody invests in their “test in prod” tooling. Firstly, what is logging then? How is this not tooling to ensure things are running smoothly? `less +F` anyone? Secondly, if you're running an aws/azure/gcp based server you now have a ridiculous amount of tooling for production diag, analytics and tracing.

Logging is tooling but it is the bare minimum and requires very little investment to do. Good logging requires a little more investment but often isn’t done. However, it still generally only provides insight into the known knowns and sometimes a known unknown, which is largely not where bugs happen. Better tooling would enable understanding and further aid reproducibility over mere observability.

Re: All of us test in production all the time (2019)

#140

Post by ratww » Mon, Jul 20, 2020, 10:51 AM UTC

pydry wrote: post #103

ratww wrote: post #75

Earlier quoted context omitted.

People don't say it directly, but all the constant complaining about the bloat and complexity of the JS ecosystem and the size of node_modules folder boils down mostly to two tools: Babel and Webpack.

When people make fun of leftpad they are not talking about babel and webpack. The bloat is a lot about the lack of a standard library in JS. Or rather, too many of the damn things. That and an inconsistent and unpredictable type system.

Babel was the largest package that was dependent on leftpad back in 2015 when the leftpad issue happened.

The package bloat issue doesn't happen with Typescript, for instance, which occupies a similar niche in the ecosystem (it's also a transpiler), or prettier.