41–50 of 58 posts
Because we’ve railed on “programmers” for so long everyone wants to be a “software engineer” which more often than not leads to equally insane architectures. No you don’t need multi-region availability. No you don’t need to rewrite this in rust. No you don’t need a multi-master db setup. No you don’t need graphql.
I don’t think one needs to plan on being Facebook scale from the beginning, but I do think there is some consideration of scalability that needs to be considered from the start. Maybe you don’t need to think 10 steps ahead but you should at least have plans for 3 or 4 steps ahead of you don’t want to have to rearchitect everything from scratch in a year. It might not be strictly necessary to containerize your app, fo…
Thing is if you want to scale to FB size you _have_ to re architecture / redesign. No one starts an application with the farm model and multiple colos.
Starting with partitioned data is a lot bigger jump. At most, I would say avoid building features that would require heroic efforts to implement in a partitioned world, but a single database can get huge these days, and maybe even huger by the time you need to scale. Chances are, 2TB of ram and 64 cores will fit your live DB needs; and if not, you'll know with time to make something else work
Kubernetes, Machine Learning, Microservices, Service-Mesh, are the things that immediately come to (my) mind from such discussions with start-up dev-teams. But somehow all of these are at least a few years old meanwhile. Could it be that the buzzword bingo we experienced in the last years has already peaked or why haven't we seen any new nonsense[1] (at least nothing that I'm aware that is not at least 24 months old)…
Machine learning may have been used and abused as a meaningless buzzword but kubernetes in particular and container orchestration in general is a well established and proven infrastructure technology. If you are not operating a monolith and you need to automatically deploy and scale your service then you have no technically-sound argument to not use this sort of infrastructure.
I'm not against k8 per se, but its complexity isn't something to ignore. It's all good if you have the people to run this but most places I know don't.
I think we have reached peak k8-hype last year when F. Campoy at GOTO suggested to wrap K8 around GO to make Go as resilient like Erlang/BEAM (see video below ...) https://www.youtube.com/watch?v=ScE9TnoWltA&t=1437s
as for ML I really think there are cool use-cases, but why would a pre-MVP start-up already be talking about hiring data scientists, ... when they don't even have any data (exception being they are an analytics start-up obviously).
this is how you get ants.
"Why try to be like others if you are condemned to being yourself" Link: https://mutterichbindumm.tumblr.com/post/92142897112/funeral...
Ideally orgs should sow the seeds of this 3-layer org (and corresponding tech architecture) design from the beginning and consciously choose to scale the investments in those layers as needed in each growth spurt.
Usually people tend to shortchange this advice by thinking they can make the same 10 people do the work in all the layers. And that may be ok when you are small. But specialization in systems vs app-frameworks vs app feature development is highly important for velocity. Also your hiring loops and the "hiring bar" has to be different for these roles. And usually the hiring pipeline depth and churn rate is different for these roles. So, creating a structure where impedances are matched correctly between input and output, so to speak, is immensely helpful for achieving high productivity and cost efficiency.
I think this article is basically right, but I want to provide a personal perspective on this: This is why I have found that I prefer working on a widely used product at one of these big companies. I find the most enjoyment in thinking through how to make products work acceptably for huge numbers of users, or how to pick the needles I need out of enormous data sets, or how to structure a codebase such that hundreds o…
But there's always YAGNI of some kind in any situation and I'm good with the kind of YAGNI that these scaling situations have.
Earlier quoted context omitted.
Thing is if you want to scale to FB size you _have_ to re architecture / redesign. No one starts an application with the farm model and multiple colos.
Starting with two colos might be a good idea -- at least at the point where you start thinking of adding a second machine for data availability, it may as well be in another colo so you have a disaster recovery option. Starting with partitioned data is a lot bigger jump. At most, I would say avoid building features that would require heroic efforts to implement in a partitioned world, but a single database can get hu…
This concept we need to go full microservices multi region right from the get go is silly and wasted time / effort.
Until your idea sells and your market grows there is no reason to concern yourself with the logistics of scale.
Earlier quoted context omitted.
> And they did "get it done", "it" was just a steaming pile of crap when it was done. "Why is this taking weeks to do - the previous guy was so much faster?" When I mean JGID, I also meant not make it a steaming pile of shit :)
> When I mean JGID, I also meant not make it a steaming pile of shit :) the tldr of my post is that people often don't know the difference. We also have grown a culture of people promoting "YAGNI", and its easier for some people to dismiss basic ideas with YAGNI. Yeah, adding 5 language translations to your project on day 1 - YAGNI. Storing passwords in plaintext? You need something more than that.
DRY up your code... unless the two code paths look similar but aren't the same.
YAGNI... unless "it" is a database backup, a load balancer, or test coverage of your signup flow.
KISS... unless your problem is complicated enough that a simple solution only implements half of what you need.
Microservices are great... if you've got more than one team and can support the operational overhead of SDN and service discovery. Monoliths are great... until you're pushing code to them every minute and they become impossible to refactor.
etc.
Two articles on this topic: You are Not Google (my favorite): https://blog.bradfieldcs.com/you-are-not-google-84912cf44afb A piece from my Notes to a Young Software Engineer that likens engineering media to a bazaar, not a mirror of the industry: Beware Engineering Media: https://www.nemil.com/on-software-engineering/beware-enginee... (Martin Fowler's YAGNI article is also great)
I don’t think one needs to plan on being Facebook scale from the beginning, but I do think there is some consideration of scalability that needs to be considered from the start. Maybe you don’t need to think 10 steps ahead but you should at least have plans for 3 or 4 steps ahead of you don’t want to have to rearchitect everything from scratch in a year. It might not be strictly necessary to containerize your app, fo…
Thing is if you want to scale to FB size you _have_ to re architecture / redesign. No one starts an application with the farm model and multiple colos.