71–80 of 114 posts
Earlier quoted context omitted.
[flagged]
Ads are just what pays the bills. The cluster is also used for all sorts of useful things, like mass communication, especially during emergencies, support groups, and providing information to help manage a pandemic. Software engineers aren't nuclear physicists or machinists. I'm not sure how you want to use them to build nuclear reactors.
Never heard of this "hyperscale" concept before. How is this any different from... scaling?
Supermarkets and super cars aren't enough. We need hypermarkets and hyper cars to be current. So now we can’t just scale, I guess that’s for trucks, so to be current you need hyperscale.
Earlier quoted context omitted.
> fixing their system for taking hosts in and out of service, so that it's extremely robust and reliable would likely pay big dividends in reliability. Facebook hosts can be robustly cycled. Of course. They've been doing this stuff for years. They've figured it out. That's not the issue. Scaling up brings about new problems. This article specifically mentions the 45 day rolling restart issue. That's not an issue when…
What exactly do you think requires cycling millions of hosts to take 45 days? That's a couple hundred hosts every five minutes across a large number of datacenters? I wouldn't expect it to be halved by optimization. I'd expect it to be an order of magnitude faster and take more like 4.5 days. I wouldn't be surprised (but I would be impressed) if they tried and got it down to a full cycle requiring one working day. Th…
I'm betting it's moreso that teaching developers to write software that tolerates draining properly (or is even able to communicate draining) is too difficult for them so they work around it.
Earlier quoted context omitted.
I don't follow. Reboot is downtime. Of course your architecture must allow for downtime if it happens, but it's lost money either way, your hardware is not doing any useful work while rebooting. So more computers you have, more money is lost. At small scale that's not significant, but at large scale that might become significant so there's more incentive to reduce downtime.
A reboot, a software deployment (kernel upgrade), server replacement, etc. are all the same process. That simplifies things dramatically. You can micro-optimize the 30s it takes to reboot a server, or you can simplify a runbook to have one process for any “deployment”. Different scenarios require different things but for most “web scale” things that need to be overprovisioned anyway, I’d take the simpler process.
I wish they mentioned how long a full deployment takes Meta using this method, that seems like an important detail to omit. > So, if you’d rather not have downtime with your servers, data centers, and clouds, follow Meta’s example and use live patching. You’ll be glad you did. Maybe if you're working at Meta's scale it makes sense... But I think most well designed services and applications should be able to get by ju…
> Maybe if you're working at Meta's scale it makes sense... But I think most well designed services and applications should be able to get by just fine with a full reboot of any single server. I feel like this should be the opposite... I don't work at Meta scale, but I do work for a CDN with 10s of thousands of servers, and everything we do is based on the idea that some machines will always be going down, because so…
If you each sever spends 7.5 minutes each month rebooting, that is 1% of all your computers wasted. If you have 10,000 servers, that’s worth 100 severs. If you have 1 million servers, that’s worth 10,000 servers. If each server costs $10,000, that’s $100 million dollars of compute capacity. You can see how that amount of lost computer capacity can start to justify spending engineering time on driving down the amount of time servers are rebooting.
Earlier quoted context omitted.
> Maybe if you're working at Meta's scale it makes sense... But I think most well designed services and applications should be able to get by just fine with a full reboot of any single server. I feel like this should be the opposite... I don't work at Meta scale, but I do work for a CDN with 10s of thousands of servers, and everything we do is based on the idea that some machines will always be going down, because so…
Edit: ignore the below numbers, I got hours and minutes confused. If you each sever spends 7.5 minutes each month rebooting, that is 1% of all your computers wasted. If you have 10,000 servers, that’s worth 100 severs. If you have 1 million servers, that’s worth 10,000 servers. If each server costs $10,000, that’s $100 million dollars of compute capacity. You can see how that amount of lost computer capacity can star…
7.5min/43,200min = 0.00017361
Where are you getting 1% of all your computers wasted per month???
Earlier quoted context omitted.
[flagged]
Ads act as a recommendation engine, helping people find solutions to their problems sometimes before they ever realize there is a solution.
there are some plattforms with relative "high" quality ads, but facebook is not one of them.
Earlier quoted context omitted.
Edit: ignore the below numbers, I got hours and minutes confused. If you each sever spends 7.5 minutes each month rebooting, that is 1% of all your computers wasted. If you have 10,000 servers, that’s worth 100 severs. If you have 1 million servers, that’s worth 10,000 servers. If each server costs $10,000, that’s $100 million dollars of compute capacity. You can see how that amount of lost computer capacity can star…
There’s 1,440 minutes per day. In a month with 30 days that’s 43,200 minutes per month. 7.5min/43,200min = 0.00017361 Where are you getting 1% of all your computers wasted per month???
Earlier quoted context omitted.
Edit: ignore the below numbers, I got hours and minutes confused. If you each sever spends 7.5 minutes each month rebooting, that is 1% of all your computers wasted. If you have 10,000 servers, that’s worth 100 severs. If you have 1 million servers, that’s worth 10,000 servers. If each server costs $10,000, that’s $100 million dollars of compute capacity. You can see how that amount of lost computer capacity can star…
There’s 1,440 minutes per day. In a month with 30 days that’s 43,200 minutes per month. 7.5min/43,200min = 0.00017361 Where are you getting 1% of all your computers wasted per month???
So the correct numbers would be 7.5 minutes of downtime divided by 43,200 minutes times 1,000,000 servers. That’s 173 servers wasted. That is probably still enough servers wasted to devote some engineering time to increasing utilization.
Earlier quoted context omitted.
I have objections to the name, but it goes under Ksplice for User Space. https://blogs.oracle.com/virtualization/post/ksplice-zero-do...
How was Oracle to work at?
I think the biggest "big company" blunder while I was there was a public blog post by someone high up at the company decrying open source as bad/wrong while at the same time, Oracle was doing all this other open source stuff.