BEAM for the win
Why is everything so scalable?
331–340 of 383 posts
Re: Why is everything so scalable?
#332Earlier quoted context omitted.
THANK YOU. People look at me like I’m insane when I tell them that their overly-complicated pipeline could be easily handled by a couple of beefy servers. Or at best, they’ll argue that “this way, they don’t have to manage infrastructure.” Except you do - you absolutely do. It’s just been partially abstracted away, and some parts like OS maintenance are handled (not that that was ever the difficult part of managing s…
I totally agree. So much complexity for generally no good reason [0]. I saw so much of this that I ended up starting a company doing the exact opposite. I figured I could do it better and cheaper, so that that's now what we do! If anyone wants to bail out of AWS et al and onto a few beefy servers, save some money, and gain a DevOps team in the process, then drop us an email (adam at domain in bio). [0] My pet theory…
> My pet theory about the real reason: the hyper-scalers hire all the engineers who have the skills to deploy-to-a-few-beefy-servers, and then charge a 10x multiplier for compute.
This is also my pet theory, and it’s maddening. They’ve successfully convinced an entire generation of devs that physical servers are super scary and they shouldn’t ever have to look at them.
Re: Why is everything so scalable?
#333>Modules cannot call each other, except through specific interfaces (for our Python monolith, we put those in a file called some_module/api.py, so other modules can do from some_module.api import some_function, SomeClass and call things that way. This is a solution to a large chunk of what people want out of microservices. There are just two problems, both of which feel tractable to a language/runtime that really wan…
Though even the solution to 1 doesn't solve "ACLs", which distribution does. If you want to ensure your module is only called by an approved list of upstream modules, public / private isn't granular enough. (You can solve it with attributes or some build tools, but it's ad-hoc and complex, doesn't integrate with the editor, etc. I've always thought something more granular and configurable should've been built into Ja…
Exactly. There should be no problem having tens of thousands of stateless, I/O bound database-transaction-wrapper endpoints in the same service. You're not going to run out of memory to hold the binary or something. If you want to partition physical capacity between groups of endpoints, you can probably accomplish that at the load balancer.
Having the latent capability to serve endpoint A in the binary is not interfering with endpoint B's QPS unless it implies some kind of crazy background job or huge in-memory dataset. Even in this case, monoliths normally have a few different components according to function: API, DB, cache, queue, background worker, etc. You can group workloads by their structure even if their business purposes are diverse.
Re: Why is everything so scalable?
#334Earlier quoted context omitted.
Working on various teams operating on infrastructure that ranged from a rack in the back of the office, a few beefy servers in a colo, a fleet of Chef-managed VMs, GKE, ECS, and various PaaSes, what I've liked the most about the cloud and containerized workflows is that they wind up being a forcing function for reproducibility, at least to a degree. While it's absolutely 100% possible to have a "big beefy server arch…
You have to remove admin rights to your admins then, because scrappy enough DevOps/platform engineers/whatever will totally hand-edit your AWS infra or Kubernetes deployments. I suffered that first hand. And it's even worse that in the old days, because at least back in the day it was expected.
Nah, just run Puppet or similar. You’re welcome to run your command to validate what you already tested in stage, but if you don’t also push a PR that changes the IaC, it’s getting wiped out in a few minutes.
I hate not having root access. I don’t want to have to request permission from someone who has no idea how to do what I want to do. Log everything, make everything auditable, and hold everyone accountable - if I fuck up prod, my name will be in logs, and there will be a retro, which I will lead - but don’t make me jump through hoops to do what I want, because odds are I’ll instead find a way around them, because you didn’t know what you were doing when you set up your security system.
Re: Why is everything so scalable?
#335Earlier quoted context omitted.
Or at least you have to automatically destroy and recreate all nodes / VMs / similar every N days, so that nobody can pretend that any truly unavoidable hand-edits during emergency situations will persist. Possibly also control access to the ability to do hand edits behind a break-glass feature that also notifies executives or schedules a postmortem meeting about why it was necessary to do that.
I know of at least one organisation that'd automatically wipe every instance on (ssh-)user logout, so you could log in to debug, but nothing you did would persist at all . I quite like that idea, though sometimes being able to e.g. delay the wipe for up to X hours might be slightly easier to deal with for genuinely critical emergency fixes. But, yes, gating it behind notifications would also be great.
Was this Mozilla?
Re: Why is everything so scalable?
#336Earlier quoted context omitted.
You’re thinking purely in terms of app performance. have you ever seen a terrible db schema? Having to suddenly iterate fast with a brittle codebase that doesnt really allow that ive seen bring teams to their knees for a year+. I’ve seen monoliths because of their sheer size and how much crap and debt is packed into them, build and deploy processes taking several hours if not an entire day for some fix that could be…
That was the reality of the fintech I worked at. The schema wasn't really a problem, but the sheer amount of queries per request. Often a user opening a page or clicking a button would cause 100-200 database queries, including updates. This would prevent strategies such as "just replicating the data somewhere". It was so badly architected that every morning the app would stop responding due to users doing their morni…
sounds so similar to a fintech situation I’ve been in once I swear I was gonna say we worked at the same place, but the size of company is wrong. I’ve now seen pretty similar things since, enough to say it’s probably everywhere
Re: Why is everything so scalable?
#337You can get an awful lot done with Go, MySQL or MariaDB, FCGI, Apache or Ngnix. You start with one server on shared hosting for a few dollars a month. Then scale up to a dedicated server, if the load appears. Then scale up to multiple servers with a load balancer. Replicate the database. On the load side, if you're accumulating "statistics" about user behavior, do you really need them for every user? Maybe only one u…
The problem with a VPS/baremetal solution is always high availability. Once you introduce HA, you may as well go into the cloud since HA'ing stateful stuff using VPS is a major pain. Also, you can get a lot done with a serverless (FaaS/PaaS) solution and a simple DB like DynamoDB.
I don't disagree with this, but I think people sometimes miss the fact that even cheap commodity hardware is way more reliable than most software. Your site may go down more often due to bugs in your HA configuration than it would have gone down because of your VPS dying once a year or so. If you are not a huge operation, the sweet spot for HA is probably something very simple (e.g. two servers behind a load balancer, or something like that).
Re: Why is everything so scalable?
#338Earlier quoted context omitted.
THANK YOU. People look at me like I’m insane when I tell them that their overly-complicated pipeline could be easily handled by a couple of beefy servers. Or at best, they’ll argue that “this way, they don’t have to manage infrastructure.” Except you do - you absolutely do. It’s just been partially abstracted away, and some parts like OS maintenance are handled (not that that was ever the difficult part of managing s…
What I say is that we massively underestimate just how fast computers are these days
If you know anything about hardware and look at the typical instances AWS is serving up (other than the ludicrously expensive ones) it's Skylake and older.
I think people have a warped perception of performance, if only because the cloud providers are serving up a shared VM on equipment I'd practically class as vintage computing. You could throw some of the same parts together from eBay and buy the whole system with less than a few months worth of the hourly on-demand cost.
Re: Why is everything so scalable?
#339Just to be honest for a bit here... we also should be asking what kind of scale? Quite a while ago, before containers were a thing at all, I did systems for some very large porn companies. They were doing streaming video at scale before most, and the only other people working on video at that scale were Youtube. The general setup for the largest players in that space was haproxy in front of nginx in front of several…
Exactly this! The educational product I work on is used by hundreds of thousands of students a day, and the secret to our success is how simple our architecture is. PHP monoliths + Cache (Redis/Memcached) scale super wide basically for free. We don't really think about scalability, it just happens. I have a friend whose startup had a super complicated architecture that was falling apart at 20 requests per second. I u…
crud app for async chat app works only that far. when you start getting a lot of customers (companies, etc), big chat rooms, etc - things get complicated.
i saw this kind of system that started as simple crud app, and many years later developers still try to resolve some of the originals sins.
Re: Why is everything so scalable?
#340Earlier quoted context omitted.
Exactly this! The educational product I work on is used by hundreds of thousands of students a day, and the secret to our success is how simple our architecture is. PHP monoliths + Cache (Redis/Memcached) scale super wide basically for free. We don't really think about scalability, it just happens. I have a friend whose startup had a super complicated architecture that was falling apart at 20 requests per second. I u…
crud app for async chat app works only that far. when you start getting a lot of customers (companies, etc), big chat rooms, etc - things get complicated. i saw this kind of system that started as simple crud app, and many years later developers still try to resolve some of the originals sins.
If it takes about long to resolve "original sins" it wasn't simple enough to begin with.