Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

All of us test in production all the time (2019)

increment.com

Post reply on HN
Print view View original

41–50 of 164 posts

Re: All of us test in production all the time (2019)

#41

Post by moreaccountspls »

burntoutfire wrote: post #20
vinceguidry wrote: post #15

A lot of this boils down to the constant churn in frameworks, instrumentation, programming languages. Developers don't have time to master one way of doing things, it's just a constant lava layer of crap. I took the time to learn how to build and maintain Ruby on Rails systems, hoping it would be the ticket to a fun, manageable career. Any project I worked on was an island of sustainable, fast development where the t…

> After it happened twice, well, might as well go into devops. My observation of the devops world is that the rotation of technologies and frameworks there is faster than in backend development.

Yes, but there at least is some genuine innovation going on. The backend frameworks are just churn for solving the same boring and lucrative business problems in new, fashionable ways.

Re: All of us test in production all the time (2019)

#42

Post by danpozmanter »

"Engineers should be on call for their own code." - Would you rather work someplace you are expected to be on call 24/7, or a company that doesn't require that?

It isn't the norm, and it isn't competitive. It's just more "always on" culture in the workplace - and that's not healthy. A company should understand workers need real breaks - and being on call is not a real break.

Re: All of us test in production all the time (2019)

#43

Post by cagenut »

KaiserPro wrote: post #39
throwlogon wrote: post #29

Earlier quoted context omitted.

> I guess stable just wasn't good enough I suspect that's meant to be rhetorical, but from a business perspective, was stable good enough? Sometimes what works now won't be competitive in the near future.

Take a new site for example Its just text and pictures on a web page. The new york times re-invented a database out of kafka to power theirs (mind bogglingly stupid) The Guardian re-wrote their CMS at least 4 times, at a cost of well over 50 million, if not more. Thats before they got to re-doing the layout. The FT spent 6(!) years rebuilding their entire stack, (150 tech staff for 6 years. a good percentage contract…

can confirm. media companies spent the last decade alternating between redesigns and cms migrations in a circular path that went nowhere while fb & goog ate not just their lunch but their breakfast and dinner too. the vast majority of it was driven by engineering-management-career resume-building and was actively detrimental to the editorial/content-production/journalism side of the house.

Re: All of us test in production all the time (2019)

#44

Post by tehlike »

bszupnick wrote: post #27
tehlike wrote: post #24

Most large companies encourage testing in prod. It's called split test. Clearly precursor to that is staging, and precursor to that is integration tests and whatnot. But none of that catches what you can with a/b test, and great monitoring.

Exactly what I was thinking, but I think that a/b test "feature" is mostly framed (and thus used?) for BI purposes as opposed to "let's see if this breaks". Don't get me wrong, where I work they definitely are open to making mistakes in production. We roll out "risky" new features with split tests and I think we use that logic well so this notion obviously exists, but I'm not sure how widespread it is.

I have seen google and Facebook first hand. The practice is widespread, but the acceptance of "let's test it in prod" as a real stage in software development varies. Some people look at you like you are crazy person that should not be trusted, some people acknowledge and know it to be true.

Feature testing is one thing, but you can go a/b testing a whole binary between new and old version to catch interesting bugs too.

Re: All of us test in production all the time (2019)

#45

Post by julianlam »

danpozmanter wrote: post #42

"Engineers should be on call for their own code." - Would you rather work someplace you are expected to be on call 24/7, or a company that doesn't require that? It isn't the norm, and it isn't competitive. It's just more "always on" culture in the workplace - and that's not healthy. A company should understand workers need real breaks - and being on call is not a real break.

I don't imagine Charity (article author) was implying that the Amazon method of being on-call for your code was ideal.

I took it to mean that code ownership is important and you should be responsible for fixing things when your code blows up

Re: All of us test in production all the time (2019)

#46

Post by moreaccountspls »

KaiserPro wrote: post #39
throwlogon wrote: post #29

Earlier quoted context omitted.

> I guess stable just wasn't good enough I suspect that's meant to be rhetorical, but from a business perspective, was stable good enough? Sometimes what works now won't be competitive in the near future.

Take a new site for example Its just text and pictures on a web page. The new york times re-invented a database out of kafka to power theirs (mind bogglingly stupid) The Guardian re-wrote their CMS at least 4 times, at a cost of well over 50 million, if not more. Thats before they got to re-doing the layout. The FT spent 6(!) years rebuilding their entire stack, (150 tech staff for 6 years. a good percentage contract…

I wish I could upvote this a thousand times. At the end of the day, it's all ego driven development. Nobody wants to think that their job has already been solved, and what's left is mundane and boring [engineering wise]. And so the business ends up with using a distributed log to handle a data set that could fit on a big thumb drive.

Re: All of us test in production all the time (2019)

#47

Post by TedDoesntTalk »

tluyben2 wrote: post #35
throwlogon wrote: post #29

Earlier quoted context omitted.

> I guess stable just wasn't good enough I suspect that's meant to be rhetorical, but from a business perspective, was stable good enough? Sometimes what works now won't be competitive in the near future.

> Sometimes what works now won't be competitive in the near future. What do you mean by that? I can understand in terms of hiring devs, this is true, but any other reason I would be curious about. Because I work with/in many companies, I see a lot of different tech and competitive is usually just what the team knows best. So unless you are making a new team, I cannot phantom why you would switch. Java with Servlets/S…

Perhaps he means that you won’t be able to attract great talent. A lot of extremely talented engineers don’t want to work on code bases and technologies that are 15 years old.

Re: All of us test in production all the time (2019)

#48

Post by chrischattin »

tluyben2 wrote: post #36
vinceguidry wrote: post #15

A lot of this boils down to the constant churn in frameworks, instrumentation, programming languages. Developers don't have time to master one way of doing things, it's just a constant lava layer of crap. I took the time to learn how to build and maintain Ruby on Rails systems, hoping it would be the ticket to a fun, manageable career. Any project I worked on was an island of sustainable, fast development where the t…

I had the same issue with Rails as I have with Node : the churn is detriment to long standing projects. Most projects I do run for years and even decades; you go run an update for security for Rails code that is 10 years old. It is a nightmare. I did not have issues like that with php, asp.net or spring.

My experience has been the opposite. The Rails ecosystem makes updates straight forward and is very well documented. Been running various Rails apps for almost a decade now.

Re: All of us test in production all the time (2019)

#49

Post by yladiz »

julianlam wrote: post #45
danpozmanter wrote: post #42

"Engineers should be on call for their own code." - Would you rather work someplace you are expected to be on call 24/7, or a company that doesn't require that? It isn't the norm, and it isn't competitive. It's just more "always on" culture in the workplace - and that's not healthy. A company should understand workers need real breaks - and being on call is not a real break.

I don't imagine Charity (article author) was implying that the Amazon method of being on-call for your code was ideal. I took it to mean that code ownership is important and you should be responsible for fixing things when your code blows up

I would argue that using the right words is important here. “On call” typically refers to a specific thing, where if something breaks that person on call is, well, called (or paged, etc.) to fix it when it happens, even at 2am. This is how I took the meaning of that paragraph as well.

If the author meant that code ownership is important, or that the engineer(s) who wrote the code are responsible, that message could have been conveyed by saying, “the persons who wrote the code should test it once it’s deployed and are responsible for fixing if it breaks or is broken when deployed.” This is much clearer and doesn’t use terms that could be understood incorrectly.

Re: All of us test in production all the time (2019)

#50

Post by chrischattin »

nthj wrote: post #33
Nbox9 wrote: post #23

Earlier quoted context omitted.

If you like Ruby on Rails so much, why don’t you find a job programming Ruby on Rails? There are so many.

My experience is that nobody actually does things The Rails Way, which is where most of the benefits come in. No, they’ll start out in Rails such that their underlying data model evolves into a huge mess (because Rails guards against most problems in the code layer, so the data model problems don’t really appear as bugs yet), and then some resume-driven developer will push for serverless or microservices to solve sca…

Rails does so much of the plumbing work for you (and does it really well) that it’s a good smell test that if you’re not working on something that the user sees, you’re probably reinventing the wheel.
Post reply on HN