11–20 of 47 posts
This is one of those posts that brings out comments about things being over architected for the business. My mind definitely went there. But, at some point, companies that want to keep talented tech people need to let them go build what they want to build. Maybe those things are over-architected for what the company needs right now, but it's tough to say if that's a bigger risk than losing talented tech people.
>> At HelloFresh we run hundreds of microservices that do everything from supply chain management and handling payments to saving customer preferences. Running microservices at scale is not without its own challenges and many companies are beginning to experience the pain of complexity. Lets solve this problem by introducing another level of indirection and not solving the root cause(?). At this point i really believ…
Having struggled with setting up kubernetes over the last month on my own, I’ve come to realize the absolute value of simplicity. In the end Helm just introduced more problems than it solved. Rather than applying configs haphazardly and relying on 3rd party services, it was ultimately much simpler to just download the configs for whatever service was needed (nginx ingress controller for me) and committing them to sou…
Earlier quoted context omitted.
I would wager that its because ex-Uber folks work there and carried their ways with them.
That's rather pointed, but also matches just about every system-design interview I've ever given to an Uber engineer. Next time, I want to ask them how they keep track of all those services! edit : replaced an incorrectly-used idiom
Computers are awesome at automating things, that goes for dev tooling as well.
If you’ve touched the ITSM space you’re used to managing and maintaining many thousand of assets. A few hundred microservices is nothing, really.
My team use what you could call a simplified CMDB (configuration management database) which is cross referenced against the service discovery.
The cmdb keep info about every service, such as persistent data-sources, vm’s etc, but most important - relationships: domain, team, services and resources.
A microservice is basically a ”ci” (configuration item) with a managed lifecycle.
Having struggled with setting up kubernetes over the last month on my own, I’ve come to realize the absolute value of simplicity. In the end Helm just introduced more problems than it solved. Rather than applying configs haphazardly and relying on 3rd party services, it was ultimately much simpler to just download the configs for whatever service was needed (nginx ingress controller for me) and committing them to sou…
The people writing posts and the people running production workloads seem to be a non overlapping ven diagram when it comes to a huge portion of kubernetes.
“Now, with our new service mesh—that only took a few months to roll out—a failure in our hello-fresh-left-pad microservice can be withstood with only a few hours of downtime” /s - Don’t go work for HelloFresh unless you hate your nights and weekends.
Earlier quoted context omitted.
That's rather pointed, but also matches just about every system-design interview I've ever given to an Uber engineer. Next time, I want to ask them how they keep track of all those services! edit : replaced an incorrectly-used idiom
Tip - use a database. Computers are awesome at automating things, that goes for dev tooling as well. If you’ve touched the ITSM space you’re used to managing and maintaining many thousand of assets. A few hundred microservices is nothing, really. My team use what you could call a simplified CMDB (configuration management database) which is cross referenced against the service discovery. The cmdb keep info about every…
Earlier quoted context omitted.
Tip - use a database. Computers are awesome at automating things, that goes for dev tooling as well. If you’ve touched the ITSM space you’re used to managing and maintaining many thousand of assets. A few hundred microservices is nothing, really. My team use what you could call a simplified CMDB (configuration management database) which is cross referenced against the service discovery. The cmdb keep info about every…
Keeping track is one thing. Actually _running_ the Lernaean Hydra of an application in production is a whole another story. The amount of "housekeeping" you have to do to keep the thing afloat is astounding: cascading failures, distributing tracing, logging and diagnostics, metrics. Even operational side of things require a lot of attention. Presumably, each microservice would require at least a minimal level of admi…
Log shipping is what we do from thousands of servers already (you should at least!), adding a shipper for a few 100 containers on a set of hosts is no big deal.
Fluent(d/bit) -> some kind of elastic? There are a few resonable patterns available that works and scales pretty well.
Failures and issues with the actual code - well I might have been lucky... DDD with somewhat senior devs where no spaghetti action takes place. The tooling we keep usually seem to pinpoint issues fairly well.
We’re on the scale of roughly 40 devs and my team of 3 support them with tooling that handles service lifecycle and operational stuff.
It let’s us be pretty fluent with what and how teams build and iterate stuff. I guess it requires a certain scale and experience though.
>> At HelloFresh we run hundreds of microservices that do everything from supply chain management and handling payments to saving customer preferences. Running microservices at scale is not without its own challenges and many companies are beginning to experience the pain of complexity. Lets solve this problem by introducing another level of indirection and not solving the root cause(?). At this point i really believ…
It’s a great way to get a uniform metrics and troubleshooting experience.
Most important though, especially if you have strategically compiled binaries as microservices, a service mesh lets you roll out improved routing logic easily because it’s all abstracted away, not encoded in client libraries in X number of languages. Same for tracing. Wanna add a field to all traces being generated? Go recompile and redeploy a 100 services... or change one service mesh config.
Other than that, envoy can usually withstand much more traffic than the service it overlays, so you can use it to provide DoS protection in depth, by limiting on service proxies everywhere. Saved us a couple outage escalations already.