Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We deploy to production over 100 times a day

monzo.com

Post reply on HN
Print view View original

11–17 of 17 posts

Re: We deploy to production over 100 times a day

#11

Post by willsewell »

Nathanba wrote: post #3

I really wonder how an HTTP application doesn't suffer from performance hits when it's based on 2000 micro services. Lets say even just 30 of those get used by a call to monzo.com. How does this not cause at least let's say ~300ms delay? I guess all the calls actually come from a local memory cache and there is almost never a real http call made to these microservices. Otherwise I have no idea how microservices are e…

This is a potential downside of this architecture. As others have already said, there are mitigations, but fundamentally each edge request is going to be more costly (time or money) to serve than having it served by a single machine/DB.

One of our engineering principals is to avoid premature optimisation, which is possibly one of the reasons our architecture has grown in this way. So far, whenever we've needed to fix a performance issue we've been able to solve it locally rather than change the architecture.

At the business level, we've been optimising for growth rather than costs, but this could change in the future, at which point we may need to reconsider our architecture. But for now it's working for us.

Re: We deploy to production over 100 times a day

#12

Post by samhw »

throw1234651234 wrote: post #7

This is comical. Your company needs 3,000 deploys a month? You deliver 3,000 features a month? Your BAs/PMs identify 3,000 useful features a month? This isn't trolling - this is a call out of the absurdity you are presenting as a positive.

Hmm, there were some absurdities when I was on the team, but I wouldn't have identified that as one of them. I'd characterise the service count as possibly slightly overboard, but not the deploy count: making changes super-incrementally was absolutely a positive thing compared with other places I've worked, although it would never work without an almost-zero-friction deployment process.

Done right, it avoids bugs - especially complex hard-to-roll-back state-related bugs - that can arise from discrete waterfall-style releases of big and complex features all at once.

Your comment seems to be making a hell of a lot of assumptions and associations which seem very specific to the environment you work in[0], but stated as though you think the entire world must be working in the same way. It feels a bit like a child insisting that they don't speak with an accent.

[0] "One deploy is one feature", "every company has 'BA's and 'PM's" [we had the latter but I'm scare-quoting both because they are far from universal], "each feature has to be 'identified' by said BA or PM", etc. Also, this seems to be written not only from a staid perspective but from a small-co perspective; for large companies, even 3000 features is not a huge number, and wouldn't be more than maybe 5 or 10 per team.

Re: We deploy to production over 100 times a day

#13

Post by Destiner »

throw1234651234 wrote: post #7

This is comical. Your company needs 3,000 deploys a month? You deliver 3,000 features a month? Your BAs/PMs identify 3,000 useful features a month? This isn't trolling - this is a call out of the absurdity you are presenting as a positive.

I think the idea is more about "avoid bundled changes that are harder to debug and roll back". A change doesn't have to be a new feature, might be a bug or typo fix.

Re: We deploy to production over 100 times a day

#14

Post by throw1234651234 »

TheCoelacanth wrote: post #9
throw1234651234 wrote: post #7

This is comical. Your company needs 3,000 deploys a month? You deliver 3,000 features a month? Your BAs/PMs identify 3,000 useful features a month? This isn't trolling - this is a call out of the absurdity you are presenting as a positive.

Why would a deploy need to be a feature or something that PMs identify? On a high-functioning team, you should have enough logging and monitoring that developers can identify many useful changes without PM involvement. A deploy could be something as minor as "I noticed an error getting logged in production, here's a one-line change to fix it" or "This operation is running slowly, here's a tweak to the query so that i…

I didn't consider bugs / performance enhancements.

Re: We deploy to production over 100 times a day

#15

Post by throw1234651234 »

samhw wrote: post #12
throw1234651234 wrote: post #7

This is comical. Your company needs 3,000 deploys a month? You deliver 3,000 features a month? Your BAs/PMs identify 3,000 useful features a month? This isn't trolling - this is a call out of the absurdity you are presenting as a positive.

Hmm, there were some absurdities when I was on the team, but I wouldn't have identified that as one of them. I'd characterise the service count as possibly slightly overboard, but not the deploy count: making changes super-incrementally was absolutely a positive thing compared with other places I've worked, although it would never work without an almost-zero-friction deployment process. Done right, it avoids bugs - e…

See comment on bugs. Our releases are "frictionless" too, but yes, I am hard-pressed to even imagine 3000 improvements a month to any major brand that comes to top of mind.

Re: We deploy to production over 100 times a day

#16

Post by samhw »

throw1234651234 wrote: post #15
samhw wrote: post #12

Earlier quoted context omitted.

Hmm, there were some absurdities when I was on the team, but I wouldn't have identified that as one of them. I'd characterise the service count as possibly slightly overboard, but not the deploy count: making changes super-incrementally was absolutely a positive thing compared with other places I've worked, although it would never work without an almost-zero-friction deployment process. Done right, it avoids bugs - e…

See comment on bugs. Our releases are "frictionless" too, but yes, I am hard-pressed to even imagine 3000 improvements a month to any major brand that comes to top of mind.

Bugs? Roll back. Bugs are generally proportional to LOC, not number of deploys - given that's a DevOps variable and not a programming one – so the only relevant difference I can imagine is less chance of a multi-bug, multi-factorial, multi-dependent, nightmarish-to-fix incident.

Re: We deploy to production over 100 times a day

#17

Post by tauchunfall »

throw1234651234 wrote: post #15
samhw wrote: post #12

Earlier quoted context omitted.

Hmm, there were some absurdities when I was on the team, but I wouldn't have identified that as one of them. I'd characterise the service count as possibly slightly overboard, but not the deploy count: making changes super-incrementally was absolutely a positive thing compared with other places I've worked, although it would never work without an almost-zero-friction deployment process. Done right, it avoids bugs - e…

See comment on bugs. Our releases are "frictionless" too, but yes, I am hard-pressed to even imagine 3000 improvements a month to any major brand that comes to top of mind.

Depends on the perspective. Adding one data field to a Kafka record, or Protobuf message, or Bigtable cell, or Postgres table is already an improvement.

There is already so much work to do consuming data from A, transforming it, and storing it into B.

Post reply on HN