71–80 of 104 posts
Earlier quoted context omitted.
> the name change to "Meta" is a big mistake I think it's too soon to tell. Facebook has really negative brand recognition (from my POV), and who knows, maybe "metaverse" style online interaction is the future. (For the record I'm anti-web3 and indifferent on metaverse communities)
Meta still redirects meta.com to https://about.facebook.com/meta I don't think it's too soon.
Some might enjoy this old Cloudflare debugging story about random crashes in production. https://blog.cloudflare.com/however-improbable-the-story-of-...
> So Google found fail-silent Corruption Execution Errors (CEEs) at CPU/cores. This is interesting because we thought tested CPUs do not have logic errors, and if they had an error it would be a fail-stop or at least fail-noisy hardware errors triggering machine checks. Previously we had known about fail-silent storage and network errors due to bit flips, but the CEEs are new because they are computation errors. While it is easy to detect data corruption due to bit flips, it is hard to detect CEEs because they are rare and require expensive methods to detect/correct in real-time.
https://muratbuffalo.blogspot.com/2021/06/silent-data-corrup...
> The paper claims that silent data corruptions can occur due to device characteristics and are repeatable at scale. They observed that these failures are reproducible and not transient. Then, how come did these CPUs pass the quality control tests by the chip producers? In soft-error based fault injection studies by chip producers, CPU CEEs are evaluated to be a one in a million occurrence, not 1 in 1000 observed at deployment at Facebook and Google... The paper also says that increased density, technology scaling, and wider datapaths increase the probability of silent errors.
Earlier quoted context omitted.
> the name change to "Meta" is a big mistake I think it's too soon to tell. Facebook has really negative brand recognition (from my POV), and who knows, maybe "metaverse" style online interaction is the future. (For the record I'm anti-web3 and indifferent on metaverse communities)
The name Meta dilutes the brand significantly. I bet if you ask people what Meta is, most people outside tech can't tell. But if you ask what Facebook is, 100% of them can. They took a really good brand name and trashed it to the point they needed to rebrand.
In a fleet of 100,000 machines, there will always be some clear failures... When the machine has 2x the number of segfaults of any other machine in the fleet, you send it for repairs and someone replaces the motherboard, ram and CPU... easy! But the painful ones are the 'subtle' failures. Why does machine PABL12 sometimes give NaN as a result while all 99,999 machines return sensible numbers? But all the burn in hard…
Was pabl12 an actual bad machine? Sounds somehow plausible, as if I heard of it before. It was an annoying struggle trying to raise the visibility of broken CPUs during my years at Google SRE. The SRE org and the rest of the software side of Tech Infra resisted the whole concept, even though it was well-known among platforms hardware eng. The process for taking a known-bad machine out of service involved 1) the machi…
If one could show that the CPU said 2+2=9, I'm sure they would yank it out right away, but "it returns 500 errors a lot" isn't very debugable. The only thing they can do is run the diag and return it to service if nothing comes up.
Earlier quoted context omitted.
How did you know it was a change at rest ? The only correct way to test for bitrot is to read the data back immediately after it was written and the cache flushed. If it's the same as the original, we know it made it to the disk undamaged. Then re-read it again after some time. If it doesn't match, re-read immediately again, ideally using a different physical memory block. Compare again. If it doesn't match, take the…
ZFS and other checksumming file systems can detect bit rot in data at rest. When data is read back, that sector is checksummed and compared against when it was written before returning the request. You can periodically scrub the entire pool to find and even fix these issues (in a pool with redundancy)
Interestingly, this site fails ungracefully (HTTP error code 500) when I try to visit from NordVPN, even after cycling through a few IP addresses. I’m noticing more and more sites block all VPN track. I get why, but it’s not good.
I've been on the other side of this (threat analysis, not Facebook). Known VPN-associated IP addresses were far more likely to be associated with abuse than average. Not just a little bit, but approaching 2 orders of magnitude worse in our case. It's not even close. It's too bad for the people who need to use public VPN services for whatever reason, but until we have perfect bot/abuse detection, banning VPN, Tor, and…
When asked, they cited possible abuse as a reason. But whitelisted them again after a while.
Interestingly, this site fails ungracefully (HTTP error code 500) when I try to visit from NordVPN, even after cycling through a few IP addresses. I’m noticing more and more sites block all VPN track. I get why, but it’s not good.
When I'm traveling I'll often pipe my traffic through a VPN on my home network. I have had some weird failures but I've usually assumed that it was due to an unreliable hotspot I'm using. Now I'm wondering if using a VPN is the real problem...
The scale at which Meta operates at really boggles my mind. I work with an ex facebook guy who was on the infra side of things and the numbers he told me.. I couldn't even imagine. And I'm working on the order of magnitude of 100m/h, but still, completely different set of challenges.
You may have posted a photo 7 years ago, and statistics show that basically nobody ever revisits it. However, in case you do, it needs to be there. So these enormous buildings do basically nothing, but still need to be there.
It makes me wonder how it can go on like this. Users only keep adding content and never remove it. The income per user cannot grow forever, storage cannot get infinitely cheap, the model has to break one day?
The scale at which Meta operates at really boggles my mind. I work with an ex facebook guy who was on the infra side of things and the numbers he told me.. I couldn't even imagine. And I'm working on the order of magnitude of 100m/h, but still, completely different set of challenges.
I once read that Facebook was opening 2 or 3 massive new data-centers in the US for the purpose of hosting stale content. You may have posted a photo 7 years ago, and statistics show that basically nobody ever revisits it. However, in case you do, it needs to be there. So these enormous buildings do basically nothing, but still need to be there. It makes me wonder how it can go on like this. Users only keep adding co…
Earlier quoted context omitted.
Was pabl12 an actual bad machine? Sounds somehow plausible, as if I heard of it before. It was an annoying struggle trying to raise the visibility of broken CPUs during my years at Google SRE. The SRE org and the rest of the software side of Tech Infra resisted the whole concept, even though it was well-known among platforms hardware eng. The process for taking a known-bad machine out of service involved 1) the machi…
Usually teams would consider a machine "bad" if that node in the cluster had elevated errors compared to the rest of the cluster they were running. Unfortunately this doesn't tell hardware teams what actually went wrong. If one could show that the CPU said 2+2=9, I'm sure they would yank it out right away, but "it returns 500 errors a lot" isn't very debugable. The only thing they can do is run the diag and return it…
And your last statement is definitely not true. I can recall multiple instances of demonstrable logic errors in which the machine repeatedly returned to service. This includes all of the machines of a certain generation of a certain vendor's CPUs that were found to have latent ALU bugs, 8 years after going into service.