How Instagram scaled to 14 million users with only 3 engineers • Hacker News Board

Re: How Instagram scaled to 14 million users with only 3 engineers

#101

Post by adamauckland » Sat, Sep 16, 2023, 9:40 AM UTC

0wis wrote: post #8

Interesting I wonder if its seems easy because it’s explained simply or if it really is simple to put in place. I want to make a clone now, just to try. At least for this inspiration, this article was well worth the read. Thanks !

It's pretty simple if your goal is using boring, stable technologies. Django/Postgres/Memcached/Redis/Nginx are all really stable.

I've built quite a few projects using almost exactly the same stack over the last 15 years. Almost all are still running, those that aren't are for business reasons not technical.

The problem is, they're not exciting The Next Thing real-time javascript somethings, so a lot of devs won't want to use it.

Re: How Instagram scaled to 14 million users with only 3 engineers

#102

Post by exitb » Sat, Sep 16, 2023, 9:41 AM UTC

kristopolous wrote: post #4

"When you choose the right people you'll need only a few. When you don't, you'll need them all"

Also, the fastest way to get rich is to win the lottery. Doesn’t mean it’s a sound investment advice for everyone.

Re: How Instagram scaled to 14 million users with only 3 engineers

#103

Post by ben_w » Sat, Sep 16, 2023, 9:41 AM UTC

Sparkyte wrote: post #13

kristopolous wrote: post #4

"When you choose the right people you'll need only a few. When you don't, you'll need them all"

I think this idea hurts the industry. Don't get me wrong. I think it is important to hire the right people, but if you don't hire enough people this 3 man job is 80-100 hours a week per for months to years. Look how X has diminished in quality as Elon started slashing team sizes. Then when you design more features, security and other various systems to serve the customers it will creep in complexity. You can not esca…

> Look how X has diminished in quality as Elon started slashing team sizes.

True but different: even ignoring how many of them were customer support and moderation, once the tech stack gets complex, you can't just snap your fingers and act like it's a simpler stack.

Even if a fresh 3-good-graduates team can reach feature parity with your now-1000-person team, when they've had 6 months from `git init` and you've been at it since the new team were in Kindergarten, the only way for the big corp to do the same is to buy out the new team and then leave them alone.

Re: How Instagram scaled to 14 million users with only 3 engineers

#104

Post by airtonix » Sat, Sep 16, 2023, 9:43 AM UTC

engineercodex wrote: post #50

surfingdino wrote: post #39

Earlier quoted context omitted.

Looks like someone may have been using ChatGPT to produce that post.

Author here. No ChatGPT was used. Just my own brain reading through old talks and articles from Instagram engineering and Excalidraw for the diagrams. I did my best to put together all the info I learned from them into a comprehensive and simple manner.

[dead]

Re: How Instagram scaled to 14 million users with only 3 engineers

#105

Post by throwaway290 » Sat, Sep 16, 2023, 9:43 AM UTC

onurcel wrote: post #79

rich_sasha wrote: post #25

Stories like this kind of make sense to me. 3 people is very few but I guess they really knew what they are doing. Meanwhile, all these orgs with essentially a CRUD app, with 1,000s of engineers..? That I never understood.

These CRUD apps need complex business rules, requiring expertise in the domain and making them configurable on the application level for customer while trying to keep the app not bloated. Scaling is not the only challenge engineers face, but somehow it's the one that is mostly praised.

They also need to respond to customer requirements, which IG never needed to do while they had no actual customers. And as soon as fun was up and IG had actual customers (spoiler alert, advertisers) what a surprise 3 devs was not enough.

They also need to quickly respond to downtime, because unlike IG if some of those CRUD apps go down in B2B world you are often losing customers actual money not just ad views

Re: How Instagram scaled to 14 million users with only 3 engineers

#106

Post by GaelFG » Sat, Sep 16, 2023, 9:45 AM UTC

thebears5454 wrote: post #72

EspressoGPT wrote: post #23

Earlier quoted context omitted.

> Look how X has diminished in quality as Elon started slashing team sizes. He had a point there though: He said that there seem to be "3 managers 'managing' one engineer", and I believe this is a common problem in the industry. VC-funded startups are terribly overstaffed and over-inflated.

There is not a single VC that is three managers to one engineer

"I won't give names but trust me", VC funded tech companies with 3-10 bros 'who have or not a vague technical background but more see themselves as high level thinkers' between founders and their first hires for 'key strategic roles' followed by an engineer and one to three interns to fill production role are a common reality, at least on startup scenes I attended. In the startup process 'marketing' is also glorified (not that i disagree it's importance) so once you have one or two guy who can make demos, hiring marketing people is often the next priority.

Re: How Instagram scaled to 14 million users with only 3 engineers

#107

Post by rapnie » Sat, Sep 16, 2023, 9:45 AM UTC

Here's recent insiders experience on developing Meta's Threads app on top of Instagram infra: https://newsletter.pragmaticengineer.com/p/building-metas-th...

> In July of this year, Meta launched its latest mobile app, Threads, a microblogging service and new rival to X, formerly Twitter. In the first five days following its launch, the app achieved 100M downloads – a new record for the company by some margin. Meta’s previous record for new app installs in the first 5 days after launch was 1M.

Built with a slightly larger team, considered an agile team: 3 product managers, 3 designers, about 60 engineers.

Re: How Instagram scaled to 14 million users with only 3 engineers

#108

Post by tiffanyh » Sat, Sep 16, 2023, 9:45 AM UTC

Threads

I find it interesting that Meta choice largely this same tech stack for their newly created Threads service.

Re: How Instagram scaled to 14 million users with only 3 engineers

#109

Post by ahoka » Sat, Sep 16, 2023, 9:55 AM UTC

Octabrain wrote: post #81

I find the simplicity of the stack brilliant. It makes me to think now, to which extent, the industry nowadays simply suffers from a mix of lack of knowledge, CV driving design and big players in the game trying to sell you overkilling solutions and approaches for their own economical benefit. Have we perhaps, fell into a big enchant and, at the end of the day, 99% of companies out there could just use a classic LAMP…

I just imagined trying to sell this architecture for a new product in an imaginary company, an amalgamation of every place I have ever worked:

You have to change to Azure, because we are Microsoft partners and we have free credit.

The credit is not too much though and we have to spend the same money on useless trainings so we keep being partners.

4 core and 8GB should be plenty for your dev VMs, that’s the largest we can run on free MSDN accounts.

Have you tried the managed API gateway? Why not?

You should use managed caching on the edge.

We already have an on-prem SQL Server, use that to cut costs. Yeah it runs on Vmware and network storage.

Do we have support contracts for Nginx, Ubuntu? We have RHEL licenses, so you should use that.

Can we run this in our OpenShift cluster instead? To cut license cost it will be co-deployed with the developer envs of other product, but just set resource limits. Yeah, we only have NFS storage.

S3? Just use a PVC, it’s the same.

We decided on Datadog for unrelated product A and B, so you must use that. The license is expensive, so only log errors please.

We use Kafka for the workqueue, but limited to 2 cpu cores to make it cheap, so please make sure not to send too many notifications.

Python is not for production things, we will assign an offshore team to rewrite it in Spring Boot. We target Java 11, because our productivity increasing libraries are not yet updated.

Minor change needed for deploy: every service should build it’s own RPM package in it’s dedicated git repository.

You need to submit the architecture diagrams and service documents next week, thanks for the meeting.

What did I miss?

Re: How Instagram scaled to 14 million users with only 3 engineers

#110

Post by nickserv » Sat, Sep 16, 2023, 9:56 AM UTC

devjab wrote: post #86

Scarblac wrote: post #11

Earlier quoted context omitted.

That's why it's important to keep the number of devs low, it makes it less likely that one starts talking about microservices.

In two decades of development I've never once seen a monolithic architecture that with some form of shared database not be terrible for the business it powered. I certainly understand why it's very common, it's what's still being taught to most CS students in my country after all, and, it's frankly a lot easier to implement. The result, however, is always the same. It ends up being a mess where nobody can do anything…

Agree with most everything you've said. Just want to point out that IT stuff not working properly is only grudgingly accepted when users are captive. Could be a government service (as in your example), corporate monopoly, or a work mandated application. If it doesn't work properly, users are stuck with it no matter what.

But for anything where there's healthy competition, this completely changes. Errors, bugs, conceptual problems, etc absolutely will have an extremely negative impact.

As an example I once worked for a company selling tickets online, but there were numerous bugs, and the system would often crash under load. Long story short, we lost many users to competitors, that company is no longer independent, and all that code is now legacy.

Compare with the monopoly situation of Ticketmaster, they are far worse than this company ever was, and are quite successful, with a large user base. That hates them ;-)