91–100 of 114 posts
I wish they mentioned how long a full deployment takes Meta using this method, that seems like an important detail to omit. > So, if you’d rather not have downtime with your servers, data centers, and clouds, follow Meta’s example and use live patching. You’ll be glad you did. Maybe if you're working at Meta's scale it makes sense... But I think most well designed services and applications should be able to get by ju…
Earlier quoted context omitted.
1x, 10x, 1000x are all "scales", yet problems may be a little bit different at each of them
I'm just quoting the wikipedia definiton of hyperscale. Nowhere in there does it say anything about 1000x, probably because that is an ill-defined concept. 1000x what? Today's computers are a 1000x the ones from the 90s, should we call them all hypercomputers? Pretty much any startup can boot 20,000 nodes on aws, are they all hyperstartups hyperscaling? Seems like marketing nonsense.
Why compare to the past capabilities, wtf?
>Pretty much any startup can boot 20,000 nodes on aws, are they all hyperstartups hyperscaling?
Now think how many nodes can Google, Microsoft, Fb, etc. run in their 10s of datacenters.
Earlier quoted context omitted.
What exactly do you think requires cycling millions of hosts to take 45 days? That's a couple hundred hosts every five minutes across a large number of datacenters? I wouldn't expect it to be halved by optimization. I'd expect it to be an order of magnitude faster and take more like 4.5 days. I wouldn't be surprised (but I would be impressed) if they tried and got it down to a full cycle requiring one working day. Th…
1 Million hosts over 45 days = 15 min per host That's a very realistic/optimistic number (especially as you do want to wait for all services to be running and marked as healthy) "Oh but you can batch this" sure, but you don't want too much of a big batch that will make your service slow or want to risk shooting yourself in the foot - like rebooting your whole control plane then figuring out it doesn't work like that…
Earlier quoted context omitted.
1 Million hosts over 45 days = 15 min per host That's a very realistic/optimistic number (especially as you do want to wait for all services to be running and marked as healthy) "Oh but you can batch this" sure, but you don't want too much of a big batch that will make your service slow or want to risk shooting yourself in the foot - like rebooting your whole control plane then figuring out it doesn't work like that…
That's still ridiculously slow. I'd expect them to have hundreds of Microservices. Each one of those should be able to handle a random restart at any point in time so they should absolutely be able to restart 100s of servers concurrently without major disruptions. Hell on Facebook scale a whole-Datacenter going down should not cause service disruptions.
Taking 45 days is probably more about caution and resolving issues systematically rather than pushing a big button and hoping you don’t cause issues.
I’d expect them to have thousands of microservices - and you only have to find a way to break one to cause big issues.
Earlier quoted context omitted.
Supermarkets and super cars aren't enough. We need hypermarkets and hyper cars to be current. So now we can’t just scale, I guess that’s for trucks, so to be current you need hyperscale.
give it a couple more years, we'll have ultrascale
Earlier quoted context omitted.
That's still ridiculously slow. I'd expect them to have hundreds of Microservices. Each one of those should be able to handle a random restart at any point in time so they should absolutely be able to restart 100s of servers concurrently without major disruptions. Hell on Facebook scale a whole-Datacenter going down should not cause service disruptions.
This does assume that nothing is getting broken along the way. Taking 45 days is probably more about caution and resolving issues systematically rather than pushing a big button and hoping you don’t cause issues. I’d expect them to have thousands of microservices - and you only have to find a way to break one to cause big issues.
Earlier quoted context omitted.
Post acquisition, we moved offices, and used their HR department, and certain product decisions were made for us, but we largely did our own thing, for better or worse, so I don't know how much I can speak about how it was to work at Oracle. We were under Wim Coekaerts who is a big open source guy. Oracle's reputation as a cutthroat legal entity is well deserved, but working at that side of the company it felt unfair…
> but working at that side of the company it felt unfair because, Oracle's open source contributions with VirtualBox, the UEK, and others are lost in the grar over MySQL. They are also remembered for closing OpenSolaris and shaking people down over the VirtualBox extension pack ( https://www.theregister.com/2019/10/04/oracle_virtualbox_mer... ).
Useful word.
I wish they mentioned how long a full deployment takes Meta using this method, that seems like an important detail to omit. > So, if you’d rather not have downtime with your servers, data centers, and clouds, follow Meta’s example and use live patching. You’ll be glad you did. Maybe if you're working at Meta's scale it makes sense... But I think most well designed services and applications should be able to get by ju…
> Maybe if you're working at Meta's scale it makes sense... But I think most well designed services and applications should be able to get by just fine with a full reboot of any single server. I feel like this should be the opposite... I don't work at Meta scale, but I do work for a CDN with 10s of thousands of servers, and everything we do is based on the idea that some machines will always be going down, because so…
There's a big, big difference, especially at the scale of hundreds of thousands / millions of servers, between designing such that your architecture can suffer 1% of servers being offline and 10% of servers being offline. If you have 1 million servers, even if you could take 1% of them offline at once (i.e. 10,000 servers), if it takes 5 minutes to reboot, you then need to wait 5 minutes * 100 one-percent-buckets = 500 minutes, or 8.3 hours to do a full patch. When you have critical security updates (like Heartbleed) you simply cannot have unpatched servers exposed to the Internet for that much time. And that's not including the amount of time it takes to actually send reboot/patch commands to 10,000 servers.
The larger the bucket, the more likely a bad patch is noticed by the public, and the more likely that an ordinary traffic spike (for which your extra capacity is ordinarily there for) will overwhelm your servers (since your extra capacity is being used to handle patch rollout and rebooting). Sure, you can plan and add even more capacity to compensate, which makes it take even longer to rollout a patch, and now Finance is knocking at your door wondering if you really need all these servers and maybe you could decommission some of them to save money.
It's a fundamentally difficult problem at hyperscaler scale.
Never heard of this "hyperscale" concept before. How is this any different from... scaling?
I dislike the term 'hyperscale' as it lacks concrete meaning. We need a metric akin to kilobytes or megabytes for storage. Terms like 'kiloscale' and 'megascale' could better indicate the scaling range of a service. For instance, scaling a service to a thousand instances and back to zero or rolling out patches to thousand instances could be termed 'kiloscale'.