Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Shopify reduced storefront response times with a rewrite

engineering.shopify.com

Post reply on HN
Print view View original

41–50 of 76 posts

Re: How Shopify reduced storefront response times with a rewrite

#41

Post by chiefalchemist »

lazyant wrote: post #33

I didn't care especially for the technical details, what I like about this article is that the first thing they mention is the success criteria of the project (hopefully it was done at the very beginning, before any implementation). Then on top of that, they created an automated tool to verify such criteria automatically and objectively. This is a great approach and unfortunately I don't think many (most?) software p…

It's not only software.

1) What is the goal? What defines success?

2) What are the KPI's? How are we going to measure it?

These are baseline questions to any endeavor of substance. Yet, they are rarely defined.

Re: How Shopify reduced storefront response times with a rewrite

#42

Post by benatkin »

cbothner wrote: post #32
pqdbr wrote: post #17

Earlier quoted context omitted.

Someone replied but deleted right when I was posting this answer, so I'm replying to myself: What I didn't understand was why the listed performance optimizations couldn't be implemented in the monolith itself and ensued the development of a new application, which is still Ruby. In a production env, the request reaches the Rails controller pretty fast. I know for a fact that the view layer (.html.erb) can be a little…

Don't forget that a Shopify store is 100% customizable by merchants using Liquid (Turing complete, not that you should try). There is no .html.erb layer. Think of Storefront Renderer as a Liquid interpreter using optimized presenters for the business models.

Liquid is designed so template authors don't have to be trusted. That's great and I wish it were more common.

Here's an example of a disclaimer that should be attached to most templating languages: https://mozilla.github.io/nunjucks/templating.html

Re: How Shopify reduced storefront response times with a rewrite

#43

Post by raihansaputra »

aloukissas wrote: post #37

Naive question: the "storefront" piece seems like it's a static page. Why does it need SSR? Even so, it could be SSR'ed to static _once_ (kind of how NextJS does this from 9.3+), then have it served by CDN/edge. I'm probably missing something here.

Throwing opinions here, but after working a bit with Shopify themes, there might be some reasons to stick with SSR rather than aggressive caching. First, the storefront can be dynamic depending on visitor region/login/logout. Second, Shopify have most of the logic on the backend, even having non-js html nodes for ordering/add to cart. Third, I don't think the visit distribution of the stores makes caching economically viable (the top 20% store probably don't account for +60% server load).

Re: How Shopify reduced storefront response times with a rewrite

#44

Post by pushrax »

notsureaboutpg wrote: post #24

Most commenters are focused on the optimizations made, but I actually think the custom routing and verification mechanism is the interesting bit. That kind of a tool could be handy in lots of scenarios (comparing the same service written in two different languages or with different dependencies, etc). But how does their verifier mechanism deal with changes in the production database between responses? If the response…

Differing inputs causing verification failures is indeed an issue. In addition to data access races, replication latency also causes this. The legacy service always reads from the primary MySQL instances per shard, but the new service always reads from replicas for scalability and geo distribution.

One slightly helpful mitigation we have in place relies on a data versioning system meant for cache invalidation. The version is incremented after data changes (with debouncing). To reduce false negatives, we throw out verification requests where the two systems saw different data versions. It's far from perfect, but it's been effective enough.

Re: How Shopify reduced storefront response times with a rewrite

#45

Post by gravypod »

Shopify has traditionally been an example people have pointed to for scaling a monolith with a large growth factor in all areas: team size, features, user base size, general "scale" of the company.

Does anyone on here, who has worked on this project or internally at Shopify, feel that this project was successful? Do you think this is the first, of a long and gradual process, where Shopify will rewrite itself into a microservice architecture? It seems like the mentality behind this project shares a lot of commonly claimed benefits of microservices.

> Over the years, we realized that the “storefront” part of Shopify is quite different from the other parts of the monolith

Different goals that need to be solved with different architectural approaches.

> storefront requests progressively became slower to compute as we saw more storefront traffic on the platform. This performance decline led to a direct impact on our merchant storefronts’ performance, where time-to-first-byte metrics from Shopify servers slowly crept up as time went on

Noisy neighbors.

> We learned a lot during the process of rewriting this critical piece of software. The strong foundations of this new implementation make it possible to deploy it around the world, closer to buyers everywhere, to reduce network latency involved in cross-continental networking, and we continue to explore ways to make it even faster while providing the best developer experience possible to set us up for the future.

Smaller deployable units; you don't have to deploy all of shopify at edge, you only need to deploy the component that benefits from running at edge.

Re: How Shopify reduced storefront response times with a rewrite

#46

Post by jashmatthews »

nicoburns wrote: post #21
sbarre wrote: post #14

Earlier quoted context omitted.

Obvious how? Are you going to restructure literally thousands of employees and their teams, staffed with Rubyists and organized around your current setup? Will you re-hire and/or re-train everyone? That doesn't seem so obvious... At the scale of a team like Shopify, refactoring to a different language is probably a non-starter.

If you have thousands of rubyists then you surely have hundreds who also know other languages? Seems to make sense to use a fast langauage for the small performance sensitive part of your codebase.

"Faster" languages often have big advantages in small benchmarks which get a lot smaller or even reverse once you're looking at whole application performance.

Mandelbrot (from CLBG) Ruby 246s NodeJS 8s Java 4s

Web (fortunes from TE benchmarks) Ruby + Roda + Sequel 51k rps NodeJS + Express 46k rps Java + Dropwizard 62k rps

Re: How Shopify reduced storefront response times with a rewrite

#47

Post by bdibs »

I’m aware that Ruby/Rails isn’t that quick, but it seems mind boggling that an 800ms server response time is considered tolerated, and 200ms is satisfying. I’ve never used Ruby in production so maybe my reference point is off and this is more impressive than I’m giving it credit for.

Re: How Shopify reduced storefront response times with a rewrite

#48

Post by mikepurvis »

sbarre wrote: post #14
bsaul wrote: post #3

Earlier quoted context omitted.

That’s also my question after reading this post. When trying to shave off milliseconds by going for a full rewrite, moving away from ruby seems like an obvious decision...at least intuitively..

Obvious how? Are you going to restructure literally thousands of employees and their teams, staffed with Rubyists and organized around your current setup? Will you re-hire and/or re-train everyone? That doesn't seem so obvious... At the scale of a team like Shopify, refactoring to a different language is probably a non-starter.

Sentry (otherwise a Python application) built their Symbolicator service in Rust because it was a better fit for the domain. Probably also because Armin Ronacher has become a fan of the language and simply wanted to [1]. Now, Sentry is like 100 employees or something, so it's obviously a way more agile organization that Shopify at 10x the size, but having more limited resources is also a reason to avoid spreading yourself too thin.

1: https://www.softwaresessions.com/episodes/rust-in-production...

Re: How Shopify reduced storefront response times with a rewrite

#49

Post by vlovich123 »

chiefalchemist wrote: post #41
lazyant wrote: post #33

I didn't care especially for the technical details, what I like about this article is that the first thing they mention is the success criteria of the project (hopefully it was done at the very beginning, before any implementation). Then on top of that, they created an automated tool to verify such criteria automatically and objectively. This is a great approach and unfortunately I don't think many (most?) software p…

It's not only software. 1) What is the goal? What defines success? 2) What are the KPI's? How are we going to measure it? These are baseline questions to any endeavor of substance. Yet, they are rarely defined.

It’s also important to remember that not everything worth doing or every “success” state you set can have KPIs defined (either actually impossible or the science may not be there yet).

Re: How Shopify reduced storefront response times with a rewrite

#50

Post by harrisonjackson »

benatkin wrote: post #42
cbothner wrote: post #32

Earlier quoted context omitted.

Don't forget that a Shopify store is 100% customizable by merchants using Liquid (Turing complete, not that you should try). There is no .html.erb layer. Think of Storefront Renderer as a Liquid interpreter using optimized presenters for the business models.

Liquid is designed so template authors don't have to be trusted. That's great and I wish it were more common. Here's an example of a disclaimer that should be attached to most templating languages: https://mozilla.github.io/nunjucks/templating.html

Seconded! I've had to do a lot of weird stuff in django to get around this for our user templates.
Post reply on HN