Live data from Hacker News

How Instagram scaled to 14 million users with only 3 engineers

engineercodex.substack.com

121–130 of 202 posts

Re: How Instagram scaled to 14 million users with only 3 engineers

#121

I'm more interested in how they got 14 million users with only 4-5 people. what marketing tactics did they use? I assume those should be milked to death by now.

WhatsApp used to refresh its app by tweaking its name in early days on Appstore so that it appears new in the algorithm to be recommended. I guess that practice is now prohibited on both the stores.

Re: How Instagram scaled to 14 million users with only 3 engineers

#122

Earlier quoted context omitted.

Instagram engineers themself wrote a bit about their backend infrastructure. One of the more important topics was how they shard the data [0] and this is also linked to in this blog post. [0]: https://instagram-engineering.com/sharding-ids-at-instagram-...

If it wasn't just an image site the potential for hotspotting would be insane! Size isn't a bad thing anymore since price has dropped exponentially since the inception of Instagram. I am positive they would use another modern technology today if it was present in the past. Fantastic read though.

The potential for hotspotting decreases with the number of inserts per second. Like if you only did 1 insert per second and timed it right you could put all of those inserts on one server, but this would likely not overload the server.

It's virtually impossible for anyone to hotspot in a meaningful way with this system.

Re: How Instagram scaled to 14 million users with only 3 engineers

#123
post #109

I find the simplicity of the stack brilliant. It makes me to think now, to which extent, the industry nowadays simply suffers from a mix of lack of knowledge, CV driving design and big players in the game trying to sell you overkilling solutions and approaches for their own economical benefit. Have we perhaps, fell into a big enchant and, at the end of the day, 99% of companies out there could just use a classic LAMP…

I just imagined trying to sell this architecture for a new product in an imaginary company, an amalgamation of every place I have ever worked: You have to change to Azure, because we are Microsoft partners and we have free credit. The credit is not too much though and we have to spend the same money on useless trainings so we keep being partners. 4 core and 8GB should be plenty for your dev VMs, that’s the largest we…

> What did I miss?

From the top of my head...

* A bunch of opinionated devs pushing for microservices because they heard that's what cool kids do now. No technical reason behind at all, just pure bias flamed by a couple of blog posts/YouTube talks they watched one evening.

* A bunch of devops with intentions of implementing "industry best practices and modern tooling ©" that will end up creating a house of cards in Terraform that nobody, themselves included, will dare to touch in six months down the road.

Am I sounding too cynical?

Re: How Instagram scaled to 14 million users with only 3 engineers

#124
post #109

Earlier quoted context omitted.

I just imagined trying to sell this architecture for a new product in an imaginary company, an amalgamation of every place I have ever worked: You have to change to Azure, because we are Microsoft partners and we have free credit. The credit is not too much though and we have to spend the same money on useless trainings so we keep being partners. 4 core and 8GB should be plenty for your dev VMs, that’s the largest we…

> What did I miss? The security team is getting weird reports from their internal nmap scanner that is constantly scanning this network. No, they don't know which of their vendor automated scanners are doing it, no they can't turn it off, so please add code to specifically ignore those scans. Our VMware cluster has plenty of compute left, but it is out of storage space because there are hundreds of VMs from other tea…

> An unrelated director somehow ended up seeing this diagram and had some "suggestions". Please implement these changes asap.

this user gets it

Re: How Instagram scaled to 14 million users with only 3 engineers

#125
post #114

Earlier quoted context omitted.

How does it make the backend more resilient than a monolith? Do you not realize you have multiple instances of a monolith or something?

Bad input crashes app, monolith fails over, other instance crashes. Full outage. Assuming proper vertical separation, this risk can be reduced by microservices.

It might just as well be increased by microservices due to fragile dependency chains and just different people working async and what not.

Re: How Instagram scaled to 14 million users with only 3 engineers

#126
post #7

> we can assume that Instagram was written using Objective-C and a combination of other things like UIKit. How can they know the internal infrastructure but have to assume the app language? Edit: So the entire piece is taken almost verbatim as-is from a couple old articles on instagram engineering blog. It might as well just redirect to: https://instagram-engineering.com/what-powers-instagram-hund... This is against…

Looks like someone may have been using ChatGPT to produce that post.

Actually, ChatGTP is very useful for fixing your writing.

Sometimes when a paragraph I write reads a little too harsh to the ear, I ask ChatGTP to rewrite it - it's still my original thought.

It's really effective, but I tend to tone it down a bit to sound like myself since the output can be too formal, dry, and "academic".

Re: How Instagram scaled to 14 million users with only 3 engineers

#127

Earlier quoted context omitted.

I think this idea hurts the industry. Don't get me wrong. I think it is important to hire the right people, but if you don't hire enough people this 3 man job is 80-100 hours a week per for months to years. Look how X has diminished in quality as Elon started slashing team sizes. Then when you design more features, security and other various systems to serve the customers it will creep in complexity. You can not esca…

I have noticed absolutely no difference with Twitter/X. I am a casual user, sure, but for me it seems to work well.

Media loading is dog slow these days. 10+ second waits for images in tweets to load.

That's the most directly obvious thing that has happened post takeover.

Re: How Instagram scaled to 14 million users with only 3 engineers

#128

I find the simplicity of the stack brilliant. It makes me to think now, to which extent, the industry nowadays simply suffers from a mix of lack of knowledge, CV driving design and big players in the game trying to sell you overkilling solutions and approaches for their own economical benefit. Have we perhaps, fell into a big enchant and, at the end of the day, 99% of companies out there could just use a classic LAMP…

I run a small startup. We use a monolith and docker and no kubernetes. The tech team is me and two young software engineers. I.e. I'm the only experienced person in the team. My engineers are super productive though and quick learners.

The thing is that with a small team, you need to focus your engineering efforts on things that add values, i.e. mostly functional improvements of your product. Everything else is a distraction. Anytime we hit a problem everything grinds to a halt until we fix the problem.

Non value adding activities like devops are things that we do on a need to have basis. I have used things like teraform in the past but I opted not to on this product. Reason: I'm not planning to take down my servers and recreate them. And when I do that anyway, it takes about half an hour. Not worth spending days/weeks automating. There's a reason companies employ full time devops engineers, it's actually a lot of work. I do most of the devops in my team. When I have time and when it adds value. Which is not often. I do it well and I keep it simple.

We deploy multiple times per day though; that's worth automating. Automating things you only do once just isn't. We might get around to getting some automation for setting up our infrastructure eventually. But it doesn't really solve a problem I have right now.

Like instagram, we can scale if we have to. We're in google cloud and we use a load balancer and a scaling group of simple vms that run the docker container. One nice thing in Google cloud is that you can create a vm and parametrize it with the docker container image and it will run it. No need to install anything. The default os on the vm is docker ready. So, that's one less thing for me to worry about: installing shit on vms and making sure that is updated. I simply build my docker containers, push them to the registry and tell the scaling group to apply a new instance template with the new container. It does the rolling restart. That's just 2 simple gcloud commands in a Github action.

Simple is essential. Easy to understand. Easy to fix when it misbehaves. Easy to explain to others. Simple ensures you don't waste time.

Re: How Instagram scaled to 14 million users with only 3 engineers

#129
post #124

Earlier quoted context omitted.

> What did I miss? The security team is getting weird reports from their internal nmap scanner that is constantly scanning this network. No, they don't know which of their vendor automated scanners are doing it, no they can't turn it off, so please add code to specifically ignore those scans. Our VMware cluster has plenty of compute left, but it is out of storage space because there are hundreds of VMs from other tea…

> An unrelated director somehow ended up seeing this diagram and had some "suggestions". Please implement these changes asap. this user gets it

The bear is sticky with honey.

https://piped.video/watch?v=i92Ws7qPTRg

Re: How Instagram scaled to 14 million users with only 3 engineers

#130

Earlier quoted context omitted.

I have noticed absolutely no difference with Twitter/X. I am a casual user, sure, but for me it seems to work well.

Anecdata for sure, but I use Twitter every day and features such as video upload just stopped working for me (failing silently), not to mention the site crashing or becoming completely unresponsive every week or so.

The entire multi-media infrastructure at Twitter is visibly creaking.
Post reply on HN