Running Istio In Production
engineering.hellofresh.com
Running Istio In Production
1–10 of 47 posts
Re: Running Istio In Production
#2Delivering pre-packaged meals to people requires _hundreds of microservices_. That is truly astounding.
Re: Running Istio In Production
#3>> At HelloFresh we run hundreds of microservices that do everything from supply chain management and handling payments to saving customer preferences. Running microservices at scale is not without its own challenges and many companies are beginning to experience the pain of complexity.
Lets solve this problem by introducing another level of indirection and not solving the root cause(?).
At this point i really believe that software architects who don't code, don't belong to this industry. If the implementers and operators are suffering, there should be a feedback channel.
Re: Running Istio In Production
#4“Now, with our new service mesh—that only took a few months to roll out—a failure in our hello-fresh-left-pad microservice can be withstood with only a few hours of downtime” /s - Don’t go work for HelloFresh unless you hate your nights and weekends.
Re: Running Istio In Production
#5Delivering pre-packaged meals to people requires _hundreds of microservices_. That is truly astounding.
It's not entirely unreasonable on its face; physical fulfillment is a complicated business domain. You might have services dealing with suppliers, customers, delivery vendors, regulatory compliance, payments, sales taxes, so on, so forth. You might have ML services to predict product availability or customer demand. Those subdomains might themselves be decomposed into API services, vendor gateways, background workers, etc. This doesn't factor in infra-related things like data stores, caches, etc.
Even if you only sipped the microservices kool-aid, I can easily see dozens of services.
Granted, it seems reasonable that a handful of monoliths could get the job done. Without having worked at HelloFresh, I'm inclined to think there's more to the story that we don't know. Maybe there's a good reason to have as many services as they do.
Re: Running Istio In Production
#6This is one of those posts that brings out comments about things being over architected for the business. My mind definitely went there.
But, at some point, companies that want to keep talented tech people need to let them go build what they want to build. Maybe those things are over-architected for what the company needs right now, but it's tough to say if that's a bigger risk than losing talented tech people.
Re: Running Istio In Production
#7Delivering pre-packaged meals to people requires _hundreds of microservices_. That is truly astounding.
It's not entirely unreasonable on its face; physical fulfillment is a complicated business domain. You might have services dealing with suppliers, customers, delivery vendors, regulatory compliance, payments, sales taxes, so on, so forth. You might have ML services to predict product availability or customer demand. Those subdomains might themselves be decomposed into API services, vendor gateways, background workers…
I would wager that its because ex-Uber folks work there and carried their ways with them.
Re: Running Istio In Production
#8Earlier quoted context omitted.
It's not entirely unreasonable on its face; physical fulfillment is a complicated business domain. You might have services dealing with suppliers, customers, delivery vendors, regulatory compliance, payments, sales taxes, so on, so forth. You might have ML services to predict product availability or customer demand. Those subdomains might themselves be decomposed into API services, vendor gateways, background workers…
I would wager that its because ex-Uber folks work there and carried their ways with them.
That's rather pointed, but also matches just about every system-design interview I've ever given to an Uber engineer.
Next time, I want to ask them how they keep track of all those services!
edit: replaced an incorrectly-used idiom
Re: Running Istio In Production
#9Delivering pre-packaged meals to people requires _hundreds of microservices_. That is truly astounding.
It's not entirely unreasonable on its face; physical fulfillment is a complicated business domain. You might have services dealing with suppliers, customers, delivery vendors, regulatory compliance, payments, sales taxes, so on, so forth. You might have ML services to predict product availability or customer demand. Those subdomains might themselves be decomposed into API services, vendor gateways, background workers…
> Maybe there's a good reason to have as many services as they do.
Or maybe we're in a tech bubble and if you want engineers, you have to acquiesce to their demands of working with the latest shiny tools while they create mountains of technical debt, because if you don't, they'll just go to another startup that allows that behavior, or they'll go twiddle their thumbs at a FAANG while banking $300k+ total comp for their 5-years of experience.
I'm not trying to say one could replicate this business with a few scripts, but it sure as hell doesn't need a service mesh that looks like a mutated SARS-CoV3.
Re: Running Istio In Production
#10Having struggled with setting up kubernetes over the last month on my own, I’ve come to realize the absolute value of simplicity.
In the end Helm just introduced more problems than it solved. Rather than applying configs haphazardly and relying on 3rd party services, it was ultimately much simpler to just download the configs for whatever service was needed (nginx ingress controller for me) and committing them to source control.
My biggest take away from the k8s community is that lots of people write terrible documentation and other people write blog posts and SO answers without actually understanding how kubernetes works under the hood.
There is still an ocean of depth in regards to k8s I don’t know yet, but I feel a lot stronger about getting the intermediate basics. I’m at a point where I’m being productive again.