The need for code freezes around high traffic events is usually a sign of very poor engineering culture, where patching extreme tech debt is much higher priority than technical excellence or product improvements that help the customers. A code semi-freeze for weeks is incredibly alarming. How much unremediated tech debt must be causing that? That is a massive red flag that Etsy engineering culture must be something l…
How Etsy Prepared for Historic Volumes of Holiday Traffic in 2020
11–18 of 18 posts
Re: How Etsy Prepared for Historic Volumes of Holiday Traffic in 2020
#12The need for code freezes around high traffic events is usually a sign of very poor engineering culture, where patching extreme tech debt is much higher priority than technical excellence or product improvements that help the customers. A code semi-freeze for weeks is incredibly alarming. How much unremediated tech debt must be causing that? That is a massive red flag that Etsy engineering culture must be something l…
An opinion no doubt informed by extensive experience in high-volume ecommerce.
Re: How Etsy Prepared for Historic Volumes of Holiday Traffic in 2020
#13Let's hope they're not trying to log in with Google on their phones, because that hasn't worked for about a year now.
Re: How Etsy Prepared for Historic Volumes of Holiday Traffic in 2020
#14The need for code freezes around high traffic events is usually a sign of very poor engineering culture, where patching extreme tech debt is much higher priority than technical excellence or product improvements that help the customers. A code semi-freeze for weeks is incredibly alarming. How much unremediated tech debt must be causing that? That is a massive red flag that Etsy engineering culture must be something l…
As far as I'm aware this is a very common practice for orgs with high seasonality. Have you worked somewhere with high traffic and high seasonality that doesn't implement code freezes around known peaks?
We don’t do code freezes except in very special, isolated cases of known failure risk. When we do need an isolated code freeze for one system, that is a “all hands on deck, this had better get fixed” kind of moment. If the larger system of most of our core services needed an extended code freeze to be safe, that would result in probably (justifiably) firing senior engineering leaders.
If you have a CI/CD system and you can’t trust normal automated deploys at any time, that is a huge problem. If this happened one year and you needed the code freeze, so be it - that’s just responsible risk assessment. If you intentionally plan the system to work this way every year that is egregiously bad engineering leadership.
Re: How Etsy Prepared for Historic Volumes of Holiday Traffic in 2020
#15Earlier quoted context omitted.
An opinion no doubt informed by extensive experience in high-volume ecommerce.
I run a machine learning team in a high traffic ecommerce company (specializing in stock photography and digital assets). We see an even bigger seasonal spike than Etsy.
If so, that's very impressive, but also very atypical. Source: I work in high-volume ecommerce, have friends and colleagues at other companies who do likewise, and no one I've talked to about it works anywhere that doesn't implement at least some controls on at least checkout-path deployments during holiday prep and through holiday proper. It isn't a tech debt thing, it's just good business sense: the time of year when people give you by far the most money is the time of year when you least want to risk causing problems for people who want to do so. The problem isn't untrustworthy systems or bad engineering culture; the problem is that anything short of perfection has a measurable impact on revenue, and humans achieve perfection with less than perfect reliability.
Even where I work, that revenue impact can easily amount to a major problem if we let a significant bug slip through. People buy our product for its own sake, and will do so from Amazon or Walmart or wherever if they can't get it direct - but our margin on sales through second-party vendors is nothing like as good as on direct sales, so we still take a hit. For Etsy, it's much worse; on sales through other storefronts, they get nothing. Too, a bad enough break during holiday will also risk some fraction of their subscriber revenue from sellers who, fed up at losing out on what would otherwise also be their highest revenue of the year, might go elsewhere.
It's easy to talk shit about tech debt and bad engineering culture, I get that. It's also poorly founded in fact, and evinces an apparently questionable grasp on some fundamentals of the business, besides. I don't know whether you have represented yourself well or poorly by creating this impression, but it is the impression you've created, and if that bothers you then I might suggest trying to do otherwise next time.
Re: How Etsy Prepared for Historic Volumes of Holiday Traffic in 2020
#16Earlier quoted context omitted.
I run a machine learning team in a high traffic ecommerce company (specializing in stock photography and digital assets). We see an even bigger seasonal spike than Etsy.
With no code chill at all? Not even for artifacts which, presumably unlike what your team works on, lie in the critical path for revenue generation aka checkout? If so, that's very impressive, but also very atypical. Source: I work in high-volume ecommerce, have friends and colleagues at other companies who do likewise, and no one I've talked to about it works anywhere that doesn't implement at least some controls on…
Re: How Etsy Prepared for Historic Volumes of Holiday Traffic in 2020
#17Earlier quoted context omitted.
With no code chill at all? Not even for artifacts which, presumably unlike what your team works on, lie in the critical path for revenue generation aka checkout? If so, that's very impressive, but also very atypical. Source: I work in high-volume ecommerce, have friends and colleagues at other companies who do likewise, and no one I've talked to about it works anywhere that doesn't implement at least some controls on…
The machine learning team is responsible for all search and discovery features that users engage with when searching our inventory, some of the highest traffic, uptime demanding services I’ve seen anywhere in my 15 year career. We also run all of the image and language processing that occurs in real time in the image editor post upload, along with a HA queue system that aggregates requests to route them to shared GPU…
Re: How Etsy Prepared for Historic Volumes of Holiday Traffic in 2020
#18Earlier quoted context omitted.
The machine learning team is responsible for all search and discovery features that users engage with when searching our inventory, some of the highest traffic, uptime demanding services I’ve seen anywhere in my 15 year career. We also run all of the image and language processing that occurs in real time in the image editor post upload, along with a HA queue system that aggregates requests to route them to shared GPU…
Okay, that makes sense. Why do you assume, or at least give the very strong impression of assuming, that no one else does anything like this? Or even just that Etsy doesn't?
That is a whole mess of giant neon red flags that something is wrong here regarding engineering for resilience and trusting your deployment system to catch errors.
Additionally, for a freeze lasting weeks, unfreezing and resuming merges from some backlog of frozen changes would likely introduce far greater risk to revenue or to customers than managing a trustworthy deployment workflow during high traffic events. I would even question the fundamental premise that this is safer or beneficial to customers at all. It seems much, much more likely to be a “cover your ass” method to prevent any possible blowback from outages due to unremediated tech debt that the weak leadership is not capable of protecting as a worthwhile priority for quarterly goals.