Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Github experiencing major service outages across all services.

status.github.com

Post reply on HN
Print view View original

61–70 of 147 posts

Re: Github experiencing major service outages across all services.

#61

Post by tarkwyn »

recursive wrote: post #19
tinco wrote: post #13

Earlier quoted context omitted.

How is that un-professional? Unicorns mean errors for github, and they report them. This is more professional reporting than most companies do.

> Unicorns mean errors for github I have no idea what this means, but I can see how the idea of coming up with an alternate language in which fantasy creatures represent things we already have established words for might seem unprofessional

I'd disagree. Having an easily identified image shown when a particular error is thrown can be really helpful when diagnosing problems, particularly when a customer / user is trying to communicate the problem with the site's support team. It's also quirky and friendly.

Unicorns are also supposed to be these rare, mythical beasts which I'd say is exactly the kind of imagery you want representing your server errors.

Re: Github experiencing major service outages across all services.

#62

Post by robomartin »

I did a bunch of work on GitHub today just before it went down. Talk about getting lucky.

I'm sure we will all learn a bunch from the post-mortem. These high-profile and very openly discussed failures are always good for learning all kinds of things.

No issues at all on how GitHub is handling it so far. Eager to learn what happened. Hitting refresh on the status page every so often. Better than watching underwater basket-weaving competition at the olympics.

Re: Github experiencing major service outages across all services.

#63

Post by zalew »

tinco wrote: post #57
zalew wrote: post #30

Earlier quoted context omitted.

.git/config [remote "bitbucket"] url = git@bitbucket.org:you/repo.git fetch = +refs/heads/*:refs/remotes/bitbucket/* $ git push bitbucket //edit after tinco's comment +refs/heads/ :refs/remotes/origin/ -> +refs/heads/ :refs/remotes/bitbucket/

That snippet overrides your remotes/origin branch with what you pull from bitbucket. This might not be what you want. Instead you probably should do: fetch = +refs/heads/*:refs/remotes/bitbucket/* Unless you really know what you're doing and are just crazy like that :)

I thought he thinks of a push and forget backup, so I didn't care. but yea, you're right.

Re: Github experiencing major service outages across all services.

#64

Post by h2s »

csense wrote: post #59
rolleiflex wrote: post #21

Well handled, minus the unicorns. Tangentially relevant: I just wish Github offered some sort of an academic plan for students, no private repo means that I cannot use Github at all not because I'm building closed-source software, but because I (obviously) can't put my assignment work for public viewing before the assignment deadline. So I've been using Bitbucket, which is fine and all, but I would have loved to be a…

If you want free Git hosting, you can DIY with Gitlab [1] if you want a web-based GUI like Github's. If you prefer command-line usage, I'd recommend Gitolite [2]. It allows you to give people access to git and only git (as opposed to git's built-in system, which requires granting ssh shell access); and it only uses one OS-level user/group regardless of how many people it's managing. Either of the above solutions are…

How fitting that one of the links you've posted is unavailable at the moment due to the very downtime that caused this discussion! Evidence of our over-reliance on GitHub in the open source community, perhaps.

Re: Github experiencing major service outages across all services.

#65

Post by bencevans »

silverbax88 wrote: post #50
msbarnett wrote: post #20

Earlier quoted context omitted.

> I would point out that both TFS and Mercurial do the same thing (as does ClearCase, but ClearCase is awwwful), but with much less memory consumption on the local PC. Not trying to add snark (I upvoted the above comment), just adding that it's not a GitHub feature. Well, no, obviously it's a feature of Distributed Version Control Systems in general. In fact, that's probably why OP wrote "This is why I love git (and…

Git for large projects is not lightweight, nor do they claim to be. In fact, that's the basis of their philosophy - that they store full versions of the files as opposed to just storing the changes because data storage is considered inexpensive. The only time Git uses pointers in when the files have not changed. I'm certainly not slamming Git for working exactly the way it was intended.

I think you may be thinking of a different VCS such as SVN as Git actually just stores the meta data (changes/deltas) and not full versions of the file in each commit.

As an example when you clone a repository, it downloaded it, then in goes through a process of 'Resolving Deltas'. This is it going through the history applying the changes/deltas to the files so that they are up to the most recent revision.

The only files that are added in full are binary files.

Some reading: http://git-scm.com/book/en/Git-Internals-Packfiles :)

Re: Github experiencing major service outages across all services.

#66

Post by csense »

h2s wrote: post #64
csense wrote: post #59

Earlier quoted context omitted.

If you want free Git hosting, you can DIY with Gitlab [1] if you want a web-based GUI like Github's. If you prefer command-line usage, I'd recommend Gitolite [2]. It allows you to give people access to git and only git (as opposed to git's built-in system, which requires granting ssh shell access); and it only uses one OS-level user/group regardless of how many people it's managing. Either of the above solutions are…

How fitting that one of the links you've posted is unavailable at the moment due to the very downtime that caused this discussion! Evidence of our over-reliance on GitHub in the open source community, perhaps.

Here I took it as evidence of Github's generous-to-the-point-of-being-ridiculously-foolish attitude toward their customers: They'll even give free public Git hosting to products that directly compete with their core business at a lower price point (free)!

Even if you Do No Evil (R), some people will still complain about you.

That being said, Git hosting would be better for everyone if Github had a bigger competitor in their niche.

Re: Github experiencing major service outages across all services.

#67

Post by ghshephard »

tinco wrote: post #52
rubynerd wrote: post #25

I'm amazed at how much this has knocked me on my arse. I first attempted to redo the README for a service I've just open-sourced, before realising Github is down. Then, I attempted to fix the company CI server (OOM errors because of Carrierwave needing more than 500MB of memory to run 1 spec in, for some unknown reason), which failed because it couldn't check out the code. After giving up on that, I attempted to inst…

Your company runs its source revisions through Github without a backup solution? Do you really put all your eggs in a basket you have no control over? I know that in theory a cloud solution should have a higher uptime than an amateuristic set up private server, but cloud solutions have a certain complexity and coherence that make them very vulnerable to these kinds of 'full failures' where nothing is in your control.…

Add up your downtime over a three year period by relying on Git (or gmail, or AWS) versus the cost of trying to engineer some local-backup system, and the downtime associated with that going awry.

Outages happens - as long as we're talking hours a year, pretty much everyone but life-safety systems, critical infrastructure, and payment/high-traffic commerce sites are probably better off just letting third-party cloud vendors manage their systems. Take the downtime and relax.

(Now, if downtime consistently gets into the 10s of hours/year, it's time to look for a new cloud provider. )

Re: Github experiencing major service outages across all services.

#68

Post by rubynerd »

tinco wrote: post #52
rubynerd wrote: post #25

I'm amazed at how much this has knocked me on my arse. I first attempted to redo the README for a service I've just open-sourced, before realising Github is down. Then, I attempted to fix the company CI server (OOM errors because of Carrierwave needing more than 500MB of memory to run 1 spec in, for some unknown reason), which failed because it couldn't check out the code. After giving up on that, I attempted to inst…

Your company runs its source revisions through Github without a backup solution? Do you really put all your eggs in a basket you have no control over? I know that in theory a cloud solution should have a higher uptime than an amateuristic set up private server, but cloud solutions have a certain complexity and coherence that make them very vulnerable to these kinds of 'full failures' where nothing is in your control.…

* Git is decentralised, if Github drops off the face of the earth, it will take the two of us about half an hour to fix it as we each have a copy of the codebase on our laptops.

* If I wanted the build server to point to our internal git mirror, I would configure it to point to our internal code mirror, but I want it to build off of Github webhooks.

* The "eggs in the basket" analogy is probably best saved for a situation where I'm dependant on a cloud service, such as Twilio.

* I would expect an amateuristic private server to have better uptime than a monolithic service such as Github because there are a lot less moving parts, and in our case, a lot less people doing things with it.

* The "company impact" of Github going down is next to nil. It's one o'clock in the morning on a Sunday, I'm eating string cheese, and I feel like being productive. The company is not paying me for this, and have encountered zero losses from it. We have a very simple mirror which we can use to push code to production if Github goes down, which we have never actually had to use.

Personally, I find the "keep a torch in every corner of every room because for 15 minutes last year the power was out" attitude to life is a bit over-rated, and you're planning for an edge case. I'd much rather remember where the torch is and learn to walk in the dark.

Re: Github experiencing major service outages across all services.

#69

Post by Cogito »

bencevans wrote: post #65
silverbax88 wrote: post #50

Earlier quoted context omitted.

Git for large projects is not lightweight, nor do they claim to be. In fact, that's the basis of their philosophy - that they store full versions of the files as opposed to just storing the changes because data storage is considered inexpensive. The only time Git uses pointers in when the files have not changed. I'm certainly not slamming Git for working exactly the way it was intended.

I think you may be thinking of a different VCS such as SVN as Git actually just stores the meta data (changes/deltas) and not full versions of the file in each commit. As an example when you clone a repository, it downloaded it, then in goes through a process of 'Resolving Deltas'. This is it going through the history applying the changes/deltas to the files so that they are up to the most recent revision. The only f…

To be fair, it uses the full blob until you pack, which is not immediately and depends on your settings. Large repositories spend a significant amount of effort determining optimum window pack sizes etc to optimise the packing of their history.

So, git can be quite heavy on disk usage, until all the loose objects are packed.

Re: Github experiencing major service outages across all services.

#70

Post by deoxxa »

kmfrk wrote: post #38
callahad wrote: post #17

Earlier quoted context omitted.

Exactly. I'm sitting here thinking "poor GitHub engineers" rather than "oh crap, how am I going to work?" (Though I do wish GitHub Issues weren't as centralized...)

Yup. I wanted to do some coding before heading off, but I guess that's off the table now.

What? That's the complete opposite of what OP was saying.

You don't need github to be online to work with your code or even your repositories so long as you haven't gone and thrown all your digital eggs in the same basket (i.e. as long as you have other remotes or at least local copies of your repositories).

Post reply on HN