Live data from Hacker News

Simple Systems Have Less Downtime

gkogan.co

191–200 of 271 posts

Re: Simple Systems Have Less Downtime

#191
post #54

Instagram was what, 12 employees when they got sold for a gazillion dollars? They all could fit into a van. Because they kept their system simple. It was (and still is) a monolith. Now imagine that they decided to go the microservices way. Multiply that team size by 10 at least. Don't solve problems you don't have.

The thinking is backwards, if you are only 10 people, there I no need to shape the system in a microservicr fashion. You just extract the pieces that need scaling. If you are 60 devs, you need to split the system so that everyone can work on it without walking on each other.

Microservices are not the only way to split a system.

Far from it. They are the most onerous, least tractable way to get what you are going for.

Re: Simple Systems Have Less Downtime

#192
post #64
post #57

Earlier quoted context omitted.

I would really love to read their codebase.

Read all about it: https://instagram-engineering.com/static-analysis-at-scale-a...

typically deploying to production around a hundred times per day

That is insane!

We make a release once a week at work and things still go wrong sometimes. I am in awe how they are able to pull this off, especially at their scale.

Re: Simple Systems Have Less Downtime

#193
post #188
post #145

Earlier quoted context omitted.

> But microservices are an example of simpler systems. Each microservice does far less than the whole monolith does. You can read all the code in ~15 minutes. Microservice usually means distributed system. A distributed system is more complex than a non-distributed system since it has to do everything the non-distributed system has to do and, additionally, handle all the distributed problems. Microservices just hide…

>Microservice usually means distributed system. A distributed system is more complex than a non-distributed system since it has to do everything the non-distributed system has to do and, additionally, handle all the distributed problems. Microservices just hide the complexity in places where people don't see them if they take a cursory look over the code, e.g. what is a function call in a monolith can be a call to a…

What assumption are you talking about?

Anyway, if your problem requires a distributed system, congratulations, you'll have to go to the top of that complexity hierarchy, and will have to solve all the problems that come with it.

That doesn't change anything about there being more problems. You just don't have any other option.

Re: Simple Systems Have Less Downtime

#194

In terms of the little guy having a saas, this is true for me...my saas basically is relying on s3 for both file storage and a json file per account to list out users allowed in the system. The only time it goes down is when s3 goes down or the occasional container restart.

I'd be interested in hearing more details here.

Are you saying that you use s3 as your only data store?

Re: Simple Systems Have Less Downtime

#196
post #54

Instagram was what, 12 employees when they got sold for a gazillion dollars? They all could fit into a van. Because they kept their system simple. It was (and still is) a monolith. Now imagine that they decided to go the microservices way. Multiply that team size by 10 at least. Don't solve problems you don't have.

In my experience you do end up having external dependencies and more than one service. You do end up breaking out some code into special instance types, (high ram for video processing or what have you). These are problems you do have and do have to solve so you might as well come up with a plan. Deploying microservices really isn't that hard once you make it routine, imo.

But what do I know? I would not have expected Instagram to run things like user login on the same instance as photo upload and processing.

Re: Simple Systems Have Less Downtime

#197

This is a very important idea. I like the shipping container metaphor. I have often used the idea of a 2x4 (as in lumber) in building houses. The humble 2x4 is such a simple product, but when put together with nails and screws becomes the cornerstone of a complex structure. It is simple and it just works for its intended purpose. I am working hard to apply this to the design of Webase [1] which is a #nocode platform…

You're overdoing these links. That's no doubt why you're getting downvoted and flagged.

It's fine on HN to post your own work (1) in places where it's relevant, as long as (2) you only do it occasionally and (3) are also participating in the community in the intended ways, submitting interesting articles and having curious conversation. But when users break these rules of thumb (and you've been breaking all three of them!) it crosses into spamming. The community is very aware of that and really doesn't like it, so it's not in your interest.

Doing this by hijacking top comments and threads about other people's projects is particularly not ok.

Re: Simple Systems Have Less Downtime

#198
Simple systems also benefit from Amdahl's Law, but with humans instead of CPUs. This is an observation I've been trying to sell others on for years and having only modest luck. When I can show them, they get it, but you have to push and push to get people there if words fail.

When the shit hits the fan, there's an 80% chance that your most senior members will find the problem. They may find it quicker in the simple system, even, and others might get there first. But that last 20% is a very long tail, and even longer in a complex system.

In a complex system, many of your members cannot participate in the triage process, because they don't know enough to know what's relevant. They slow down the people trying to work the problem trying to learn new things (good) or offering low-probability scenarios (bad).

There is no 'All Hands on Deck' scenario for the complex system. To be responsive, you have to kick some or even a bunch of people out of the room, and once they leave they can't really participate.

The first phase of a triage is getting a tight repro case. Many avenues are blocked until that happens. And some people have a knack for repro cases but aren't so great at debugging. With more people you get a higher quality repro, which cuts a lot of time off the rest of the process.

Part of debugging is the cost of the verification versus the likelihood. Simple Systems afford the opportunity for people to test out unlikely but plausible scenarios that are in the long tail, without distracting from the more 'boring' checks already being done.

And when bugs are identified prior to deployment, a simple system means you can hand the repro case to the responsible party and expect/demand that they get their fix on the first try.

Re: Simple Systems Have Less Downtime

#199

Earlier quoted context omitted.

WhatsApp, similarly, had 30+ employees when they got acquired [0]. They built the fastest IM on the market with 450M+ users sending 1B+ messages everyday, and at one point surpassed Facebook in terms of number of images uploaded. The engs they had were world-class, so really, I think, saying microservices (or latest-fad) get in the way etc is disingenuous since you also require world-class talent to begin with (if yo…

They chose Erlang. A language built for communication and managing wire protocols at scale. Which describes WhatsApp itself. That was probably the biggest impact single decision for WhatsApp technically.

It's amazing how much effort it takes to do something with the wrong tools.

Sunk Cost Fallacy tends to fight any broad-stroke improvements. Until a competitor starts eating your lunch.

Re: Simple Systems Have Less Downtime

#200
post #54

Instagram was what, 12 employees when they got sold for a gazillion dollars? They all could fit into a van. Because they kept their system simple. It was (and still is) a monolith. Now imagine that they decided to go the microservices way. Multiply that team size by 10 at least. Don't solve problems you don't have.

The thinking is backwards, if you are only 10 people, there I no need to shape the system in a microservicr fashion. You just extract the pieces that need scaling. If you are 60 devs, you need to split the system so that everyone can work on it without walking on each other.

I believe that the instinct to over-engineer is based in part on bad prior experiences with trying to separate concerns after it's 'too late'. Either your own personal experiences, or those of your mentors.

Lacking any better skills to identify and avoid those problems when they begin, they try to stop it from happening in the first place. Fences get erected everywhere in case they might be needed, and they frequently turn out to be in not quite the right spot or shape. The code becomes coupled to the bad interface instead of to other code, and the fixes are just as bad.

YAGNI in theory is about trying to develop those other skills, but gets twisted into an excuse for bad tech debt loads.

Post reply on HN