Live data from Hacker News

How Instagram scaled to 14 million users with only 3 engineers

engineercodex.substack.com

181–190 of 202 posts

Re: How Instagram scaled to 14 million users with only 3 engineers

#181

Earlier quoted context omitted.

A monolith doesn't force a single database. A monolith doesn't force a single process. IPC is still simpler and cheaper than network calls. A monolith doesn't force never having an external service for a specialized use case, or FFI.

> IPC is still simpler and cheaper than network calls. I specifically called out the extra complexity of network calls in microservices, not sure if you read the full comment. > A monolith doesn't force a single process I'm not convinced; if my small/specific code has it's own process, I would say it's a microservice. Sure, we can have replicas for redundancy, that doesn't mean I won't have reliability issues when my…

> I specifically called out the extra complexity of network calls in microservices, not sure if you read the full comment.

Calling out networking doesn't preclude me from mentioning IPC. IPC isn't limited to network calls, it can be as simple as shared memory and hit millions of OPS: github.com/OpenHFT/Chronicle-Map

> I'm not convinced; if my small/specific code has its own process, I would say it's a microservice.

And you'd be wrong. A core tenant of microservices is being able to individually deploy your microservices. If I spin up a new process for some high risk, highly memory intensive process I've introduced a fraction of the operational complexity of a seperate server and retained the core value proposition of reducing its blast radius if things go south.

Of course again, if you're having so much trouble handle writing software that's reliable that you being to consider isolating instability as a top benefit from your IPC setup instead of a tiny value add... it might be a sign you're not ready for microservices.

_

> True, sadly it doesn't usually work this way. People take the path of least resistance.

> Also once you add multiple DBs you start to get into eventual consistency; which is one of the harder parts of microservices.

You're making my point: If you don't have the engineering chops as a team to make a robust monolith, you definitely don't have the skills and resources to start looking at microservices.

Eventual consistency is not inherent to having multiple databases. If I have an oft changing ephemeral set of data that only affects one feature and it's creating an impedance mismatch with our main datastore, nothing is stopping us from pulling in Redis for all the queries we were previously sending to Postgres, and as far as anything relying on that feature is concerned, nothing at all changed.

With even half decent engineering, Redis going down doesn't break any differently than it would have for a microservice: you define the same error boundaries as before and the failure case ends up the same.

I mean seriously, if your team can't handle having a second data store, imagine the bedlam when you're trying to handle multiple languages across multiple data sources in a non-centralized manner?

_

Microservices are a pattern for companies where a "microservice" gets the kind of development and devops support that would justify spinning off a new mid-sized enterprise.

When you're Netflix your `api/movies/[movieId]/subtitles` endpoint is serving the kind of traffic most companies will never see in their lifetime and needs optimizations that maybe 100 companies in the world will ever need.

For the rest of us EC2 has 224C/488T CPU 24,000 GB RAM machines with 38 GBPs I/O bandwidth. If your business ever scales so far that you outgrow that, throw some of that X Billion dollar valuation money at the problem and build your microservices.

Re: How Instagram scaled to 14 million users with only 3 engineers

#182

Earlier quoted context omitted.

> IPC is still simpler and cheaper than network calls. I specifically called out the extra complexity of network calls in microservices, not sure if you read the full comment. > A monolith doesn't force a single process I'm not convinced; if my small/specific code has it's own process, I would say it's a microservice. Sure, we can have replicas for redundancy, that doesn't mean I won't have reliability issues when my…

> I specifically called out the extra complexity of network calls in microservices, not sure if you read the full comment. Calling out networking doesn't preclude me from mentioning IPC. IPC isn't limited to network calls, it can be as simple as shared memory and hit millions of OPS: github.com/OpenHFT/Chronicle-Map > I'm not convinced; if my small/specific code has its own process, I would say it's a microservice. A…

> Calling out networking doesn't preclude me from mentioning IPC.

You made the same point I made as though it was in contradiction to what I said. Adding a network call adds complexity, yes.

> A core tenant of microservices is being able to individually deploy your microservices.

And why would you not want this to be independently deployable?

> You're making my point: If you don't have the engineering chops as a team to make a robust monolith, you definitely don't have the skills and resources to start looking at microservices.

Firstly, you never made that point. Also, I never argued against it, in fact I agree completely.

> Microservices are a pattern for companies where a "microservice" gets the kind of development and devops support that would justify spinning off a new mid-sized enterprise.

Disagree, netflix has >1000 microservices.

Re: How Instagram scaled to 14 million users with only 3 engineers

#183
post #122

Earlier quoted context omitted.

If it wasn't just an image site the potential for hotspotting would be insane! Size isn't a bad thing anymore since price has dropped exponentially since the inception of Instagram. I am positive they would use another modern technology today if it was present in the past. Fantastic read though.

The potential for hotspotting decreases with the number of inserts per second. Like if you only did 1 insert per second and timed it right you could put all of those inserts on one server, but this would likely not overload the server. It's virtually impossible for anyone to hotspot in a meaningful way with this system.

Nah this will totally be an issue. You can be on the extreme end of replication or the extreme end of sharding and experience performance problems. Sharding is more likely to hotspot depending on where hot data is consistent.

The solution in most cases is a simple database that acts as a pointer database user db -> user's db. That is generated on the creation of a user.

From here you create some simple cold storage models ( if user isn't active ) and some warm models which will scale out the db if the user's db grows it shards and replicates for more read access. But the last thing you want is to slow replication or have one DB that can't move to balance resource utilization. There are some new DB tech that does this without even sweating the deets.

Re: How Instagram scaled to 14 million users with only 3 engineers

#184
post #60

Earlier quoted context omitted.

Depends on the purpose of the application though. Monolithic is a good architecture when you have a few purposeful features and functions. But when your design relies on many services to provide a wide variety of features you need to break out this design to allow teams to operate independently. Mini monoliths are more popular today than traditional monoliths of the old.

You can split things up you don't have to though. Teams operating independently is fairly orthogonal to this.

Yeah no I get you. You just want a monolith to be purposeful when you design one. Not multi-purposeful. This is also at the limitation of a programming language. I am kind of kubernetes guy, but I am dying because it relies heavily on a virtualize network distributed. It would drastically increase the performance if kubernetes clusters were built like monoliths and each kubernetes node handled traffic independently. So of like keeping it all in the same rack. Only leaving the rack if needed. But I keep seeing bad technology decisions repeated over and over. I stopped pushing because some person with a bigger title would say this is good design. Big kubernetes clusters eventually fail. Multiple small clusters survive.

Re: How Instagram scaled to 14 million users with only 3 engineers

#185
post #86

Earlier quoted context omitted.

That's why it's important to keep the number of devs low, it makes it less likely that one starts talking about microservices.

In two decades of development I've never once seen a monolithic architecture that with some form of shared database not be terrible for the business it powered. I certainly understand why it's very common, it's what's still being taught to most CS students in my country after all, and, it's frankly a lot easier to implement. The result, however, is always the same. It ends up being a mess where nobody can do anything…

The problem with most companies and monoliths is they broke the first rule of engineering, keep it simple. A tool or service should have one purpose in mind. Mutli-tools are fine if they are used infrequently, but not one tool should share the same burden or it loses efficiency.

Same thing happens in microservices too. You just need good planning an organizing.

Re: How Instagram scaled to 14 million users with only 3 engineers

#186

Earlier quoted context omitted.

> I specifically called out the extra complexity of network calls in microservices, not sure if you read the full comment. Calling out networking doesn't preclude me from mentioning IPC. IPC isn't limited to network calls, it can be as simple as shared memory and hit millions of OPS: github.com/OpenHFT/Chronicle-Map > I'm not convinced; if my small/specific code has its own process, I would say it's a microservice. A…

> Calling out networking doesn't preclude me from mentioning IPC. You made the same point I made as though it was in contradiction to what I said. Adding a network call adds complexity, yes. > A core tenant of microservices is being able to individually deploy your microservices. And why would you not want this to be independently deployable? > You're making my point: If you don't have the engineering chops as a team…

Ah sorry, I guess replying to people supporting microservices by calling out the gaps in technical knowledge they're using to justify microservices is not the same as saying ..."you definitely don't have the skills and resources to start looking at microservices"

Ah, wait it is.

> And why would you not want this to be independently deployable?

Because FAANG has more engineers devoted to managing deployment/observability/version skew/DX/scaling/security than you have engineers. Simplifying your needs in those realms helps you greatly.

And to top that off, it 100% can be independently deployable if it's a big enough separate concern: that's just SOA without the 90's XML/SOAP/RPC spin that was ESB: https://aws.amazon.com/compare/the-difference-between-soa-mi...

_

> Disagree, netflix has >1000 microservices.

That says exactly nothing. At Netflix scale their most random "trivial" endpoints are easily doing scale that entire SMEs won't ever deal with.

When FAANG is your case study in any technical discussion in a public forum, you're default wrong. I work at an AV company, I'm not about to start telling people the insane architecture we need to support ingesting petabytes of data is something that anyone else needs.

Any useful technical discussion needs to be grounded in what the 99% need, and microservices are not it.

Re: How Instagram scaled to 14 million users with only 3 engineers

#187

Threads I find it interesting that Meta choice largely this same tech stack for their newly created Threads service.

It’s not that they made the same choices again so much as they just re-used Instagram code and infrastructure.

They had the choice of using FB/Hack stack or IG/Python stack, and they choose IG/Python … which is the interesting thing.

Re: How Instagram scaled to 14 million users with only 3 engineers

#188
post #60

Earlier quoted context omitted.

You can split things up you don't have to though. Teams operating independently is fairly orthogonal to this.

Yeah no I get you. You just want a monolith to be purposeful when you design one. Not multi-purposeful. This is also at the limitation of a programming language. I am kind of kubernetes guy, but I am dying because it relies heavily on a virtualize network distributed. It would drastically increase the performance if kubernetes clusters were built like monoliths and each kubernetes node handled traffic independently.…

Replacing function call with network call does not really solve any org issues. There is pretty much 0 difference if teams are shipping "modules" for a monolith vs a microservice outside of much simplified CI/CD setup in case of monolith. You can gain some scaling efficiencies from scaling services independently but it's a minor advantage for most projects.

Re: How Instagram scaled to 14 million users with only 3 engineers

#190

Earlier quoted context omitted.

> your account looks like a bot That's not a bot if you are not a bot? I know it's 'normal', but I still it as a bug. AI classifying me as something I not is a bug.

That’s exactly what a bot would say.

Doesn’t make blocking innocent accounts automatically less of a bug.
Post reply on HN