141–150 of 198 posts
Does Github run anything like Netflix Simbian Army against it's services? As a company by engineers for engineers with the scale that github has reached, I'm a bit surprised they are lacking a bit more redundancy. Though they may not need the uptime of netflix, an outage of more than a few minutes on github could affect businesses that rely on the service.
While this was happening at Github, I noticed several other companies facing that same issue at the same time. Atlassian was down for the most part. It could have been an issue with the service github uses, but they won't admit that. Notice they never said what the firmware issue was instead blaming it on 'hardware'.
I think they should be transparent with people about such vulnerability, but I suspect they would never say so because then they would lose revenue.
Here on my blog I talked about this issue: http://julesjaypaulynice.com/simple-server-malicious-attacks...
I think it was some ddos campaign going on over the web.
Tell us which vendor shipped that firmware, so everyone else can stop buying from them.
Earlier quoted context omitted.
Git is a distributed version control system. Github is simply a place to host a repository and some issues. There is nothing stopping anyone from pushing to another remote hub for redundancy. So you want Github to open source where they put your git repo and issues? Who cares about that? It's unimportant because regardless they're still the central endpoint to many open source projects, opened or closed source. If yo…
From a developers point of view, you're right. But there are package management systems and other stuff depending on being able to download from Github. Ofc, Github isn't to blame for this, rather the ones that thought Github would be great to use as a CDN.
Rust program building, for example, seems to require that Github be up.
TL;DR: "We don’t believe it is possible to fully prevent the events that resulted in a large part of our infrastructure losing power, ..." This doesn't sound very good.
I was recently involved in an outage that occurred because the sama datacenter was hit by lightning three times in a row. Everything was redundant up the wazoo and handled the first two hits just fine, but by the time the power went out for the third time within N minutes, there wasn't enough juice left in some of the batteries!
Now would it be possible to build an automated system that can withstand this? Probably. But would your time & money be better spend worrying about other failure modes? Almost certainly.
Earlier quoted context omitted.
If you have a support agreement with them then file a ticket requesting better customer communication and link back here as an example of how to do it right. I think everyone complains in forums and online but doesn't actually file tickets about it. These things are worth tickets too.
I take it you have no experience filing tickets with them. A typical ticket goes something like this: 1. File ticket. 2. Wait. Then wait some more. Even if you pay big money for a support contract, they take a long time to respond (often > 1 hour). 3. Get a response from a first level rep who has no access to anything, has little dev experience, and asks some inane questions which I'm convinced is a purposeful stalli…
One example: redshift. Had an expensive temporary cluster that couldn't be deleted, for days. Was stuck "pending" or "rebuilding". Assigned account rep would take forever to respond, and just didn't understand, would forward directions to using AWS console. Yeah, DOESN'T WORK. After a week decided to try getting attention on twitter, got it fixed in about 12 hours.
Earlier quoted context omitted.
If you have a support agreement with them then file a ticket requesting better customer communication and link back here as an example of how to do it right. I think everyone complains in forums and online but doesn't actually file tickets about it. These things are worth tickets too.
I think a lot of folks feel that it's a useless endeavor, so they don't bother. Amazon's been operating this way for years, and they're quite a large company; it seems unlikely to me that fundamental change can happen inspired by customer tickets, even if you're paying for support. Basically, if Netflix isn't the source of the complaint, they're not going to give two fucks. /me suspects that netflix engineers get out…
This was the post I was googling for
http://techblog.netflix.com/2015/10/flux-new-approach-to-sys... (prepare to have your mind blown)
And what I found in the Google results
From 2015-04 http://techblog.netflix.com/2015/04/introducing-vector-netfl...
And 2014-01 http://techblog.netflix.com/2014/01/improving-netflixs-opera...
That is some crazy fast innovation there
Anyone have a link to a description of the firmware bug that caused the disk-mounting failure after power was restored?
A failed/failing drive present during cold boot could cause the controller to believe there were no drives present. To add insult to injury, on early BIOS versions this made the UEFI interface inaccessible. The only way to recover from this state was to re-seat the RAID controller.
There were also two bizarre cases where the operating system SSD RAID1 would be wiped and replaced with a NTFS partition after upgrading the controller firmware (and more) on an affected system (hanging/flapping drives). Attempts to enter UEFI caused a fatal crash, but reinstall (over PXE) worked fine. BIOS upgrade from within fresh install restored it.
From the changelog:
Fixes:
- Decreased latency impact for passthrough commands on SATA disks
- Improved error handling for iDRAC / CEM storage functions
- Usability improvements for CTRL-R and HII utilities
- Resolved several cases where foreign drives could not be imported
- Resolved several issues where the presence of failed drives could lead to controller hangs
- Resolved issues with managing controllers in HBA mode from iDRAC / CEM
- Resolved issues with displayed Virtual Disk and Non-RAID Drive counts in BIOS boot mode
- Corrected issue with tape media on H330 where tape was not being treated as sequential device
- resolved an issue where Inserted hard drives might not get detected properly.Earlier quoted context omitted.
This is not only obviously true, I think it is also a completely reasonable calculus. They just proved that if the entire Redis cluster goes down they can get it back in 2.5 hours. It's almost certainly a caching layer, so there is no permanent data loss. If they fix the application bootstrap dependency on a Redis connection, and they add monitoring to more easily see in the future when the Redis cluster is the probl…
>It's almost certainly a caching layer, so there is no permanent data loss. People who use Redis rarely end up using it solely as a caching layer. It often also takes on the role of an RPC facilitator and pseudo-database. GitHub's post also mentions that their engineering team had to replicate Redis' dataset before they could get the alternative hardware running, which implies that they do need some data in there bef…
I seriously doubt this version of the story. While it's possible for several hardware/firmware to fail in all your datacenters, for them to fail at the same time is highly unlikely. This may just be a PR spin to think they're not vulnerable to security attacks. While this was happening at Github, I noticed several other companies facing that same issue at the same time. Atlassian was down for the most part. It could…