GitHub major issues - repos have "lost" commits and site is erroring • Hacker News Board

Re: GitHub major issues - repos have "lost" commits and site is erroring

#61

Post by erikpukinskis » Sun, Jan 30, 2011, 1:41 PM UTC

avar wrote: post #50

erikpukinskis wrote: post #38

Earlier quoted context omitted.

It's amazing to me that you feel more confident in your own abilities to keep a server running than GitHub... you must be a really great sysadmin! The GitHub folks are pretty hardcore! I, for one, am a positively mediocre sysadmin. And the amount of time I have to spend waiting for GitHub to fix bugs is almost certainly far, far, FAR less than the amount of time I'd spend scrambling (but not waiting!) to fix my own.…

> The GitHub folks are pretty hardcore! What gives you that idea? I run several cronjobs that deal with their service, and it's down a lot compared to other similar services. My jobs deal with that gracefully, but if I wanted to I could easily set up a more reliable service than GitHub if my only goal was to host *.git repositories, not have any of the other nice things they provide.

Well, this: https://github.com/blog/530-how-we-made-github-fast

And the fact that they have somewhere north of 5 terabytes of active data, and their job queue is handling many hundreds of thousands of jobs a day, and it pretty much all happens extremely fast with nary a hiccup. I know there are outages, but I'm pushing up changes to them something like 10 hours a day every day, and I've never noticed them.

Re: GitHub major issues - repos have "lost" commits and site is erroring

#62

Post by windsurfer » Sun, Jan 30, 2011, 2:33 PM UTC

erikpukinskis wrote: post #38

windsurfer wrote: post #7

Earlier quoted context omitted.

At the price and size of Linode instances, it's cheaper and I feel better knowing I'm in control when I run Redmine+Gitosis. When I had my github account and something went wrong, I couldn't just ssh in and fix it. All I could do is wait. Now, for some people, I can see how waiting is an easier solution. I personally just don't like it. (of course, this is reffering to private repos. public repos are still king on gi…

It's amazing to me that you feel more confident in your own abilities to keep a server running than GitHub... you must be a really great sysadmin! The GitHub folks are pretty hardcore! I, for one, am a positively mediocre sysadmin. And the amount of time I have to spend waiting for GitHub to fix bugs is almost certainly far, far, FAR less than the amount of time I'd spend scrambling (but not waiting!) to fix my own.…

I don't feel like I could keep Github running as well as most, but for the entire time I've had my personal server running, it only went down once due to a hardware failure on Linode's end. For simple stuff, a simple sysadmin will suffice. Github has thousands of users and cares about performance, I have maybe 5 and I couldn't care less if my server used 2000% more cpu than it's supposed to. Different levels of complexity, you see.

Re: GitHub major issues - repos have "lost" commits and site is erroring

#63

Post by deno » Sun, Jan 30, 2011, 2:59 PM UTC

roel_v wrote: post #53

jshen wrote: post #43

Earlier quoted context omitted.

The beauty os a dvcs is that I have a full copy on my machine as well. But seriously, if you use git over ssh there is no sys admin work other than running apt-get upgrade

Maybe for a few months. When you run a server for 2 or 3 years the distro goes out of support, software upgrades are getting behind, before you know it you are compiling patches from source, ... I've done this several times over the last 5 years and am now finally moving everything to specialized hosting services. Just hosting a web site, photo repo, mail and svn on a machine (vps) is enough to make it a serious hass…

> Maybe for a few months. When you run a server for 2 or 3 years the distro goes out of support, software upgrades are getting behind, before you know it you are compiling patches from source, ...

You can get 5 years security updates support with Ubuntu Server LTS and you can set up safe unattended updates[1] with email notification if anything ever requires your attention. If you set your backups right as well, you can get away even with catastrophic hardware failures.

Remember, this is not some gigantic scale, it shouldn't be that difficult.

[1] https://help.ubuntu.com/10.04/serverguide/C/automatic-update...

Re: GitHub major issues - repos have "lost" commits and site is erroring

#64

Post by roel_v » Sun, Jan 30, 2011, 3:27 PM UTC

deno wrote: post #63

roel_v wrote: post #53

Earlier quoted context omitted.

Maybe for a few months. When you run a server for 2 or 3 years the distro goes out of support, software upgrades are getting behind, before you know it you are compiling patches from source, ... I've done this several times over the last 5 years and am now finally moving everything to specialized hosting services. Just hosting a web site, photo repo, mail and svn on a machine (vps) is enough to make it a serious hass…

> Maybe for a few months. When you run a server for 2 or 3 years the distro goes out of support, software upgrades are getting behind, before you know it you are compiling patches from source, ... You can get 5 years security updates support with Ubuntu Server LTS and you can set up safe unattended updates[1] with email notification if anything ever requires your attention. If you set your backups right as well, you…

"...it shouldn't be that difficult"

Famous last words...

I'm not saying it can't be done, I'm just saying that my experience has been different. Another example, I used to have my own installation for our websites and bug tracking and time sheet management software, on a locally-hosted machine with apache and mysql and a bunch of other software. The number of hours I've spend migrating data between versions, tweaking mod_rewrite rules, setting up backups for various databases and other data, making usage statistics etc. - I don't even want to think about it.

Recently I moved to a shared hosting server. The yearly cost of that is covered with one hours of my hourly rate (and I'm not even expensive). I can set up 50 or so different applications with a few clicks in a web UI, and upgrades are handled by it, too. Backups are taken care of, I have a web ui for dns, ssl, everything. It was like a breath of fresh air.

To each his own and each situation is different, of course. But there's something to be said for division of labor.

Re: GitHub major issues - repos have "lost" commits and site is erroring

#65

Post by mnazim » Sun, Jan 30, 2011, 4:09 PM UTC

windsurfer wrote: post #62

erikpukinskis wrote: post #38

Earlier quoted context omitted.

It's amazing to me that you feel more confident in your own abilities to keep a server running than GitHub... you must be a really great sysadmin! The GitHub folks are pretty hardcore! I, for one, am a positively mediocre sysadmin. And the amount of time I have to spend waiting for GitHub to fix bugs is almost certainly far, far, FAR less than the amount of time I'd spend scrambling (but not waiting!) to fix my own.…

I don't feel like I could keep Github running as well as most, but for the entire time I've had my personal server running, it only went down once due to a hardware failure on Linode's end. For simple stuff, a simple sysadmin will suffice. Github has thousands of users and cares about performance, I have maybe 5 and I couldn't care less if my server used 2000% more cpu than it's supposed to. Different levels of compl…

+1 to you for that. We tried github for some of our private projects, but soon switched to a linode+redmine+git/hg. Now we are no rails champs; we all are python guys but setting up redmine(with some plugins was a cake walk) same goes for git/hg. We never needed to hire hardcore sysadmins for this stuff.

PS: We love github very much. its made it so easy to collaborate everyone.

Re: GitHub major issues - repos have "lost" commits and site is erroring

#66

Post by saurik » Sun, Jan 30, 2011, 4:36 PM UTC

wfarr wrote: post #60

saurik wrote: post #47

Earlier quoted context omitted.

This comment is totally unrelated and is itself a strawman. Yes, it is easier to use GitHub: I will not argue that fact. However, using GitHub will cause people to be pulling from GitHub, and GitHub may go down. This is a tradeoff, and is one people use a lot: you use a shared platform and give up control of the URL to get easier outsourced hosting. But to argue that git's decentralization solves that problem is disi…

Web services go down. It happens. You make the choice to use them anyway. In this case, Github is very responsive about outages and clearly strives to eliminate or reduce them as much as possible.

And sometimes web services go down for good. Again, this is an understood tradeoff, and I'm not arguing that. What I do argue with is that "git is distributed" does not cancel out this particular tradeoff, which I'd the statement that was made by the person I am responding to. An actual solution used by many other services is "let me use my own hostnames with this service", which GitHub does not support for your repository, as While their fundamental value comes from the social features and nifty git UI, they seem to mentally be stuck in a "we are the git hosting company" mindset.

Re: GitHub major issues - repos have "lost" commits and site is erroring

#67

Post by icefox » Sun, Jan 30, 2011, 4:39 PM UTC

antirez wrote: post #52

Suggestion for creating a startup: For private repositories gitolite is really awesome, but I think there should be a company about it. What I mean is that github or some other company should offer hosted gitolite instances, in the following form: You pay something like price_of_small_linode_instace+price_of_service, to get: A virtual machine with gitolite up and running, with a good web interface and online document…

What do people really think of this idea? Would you buy it? I have written a web interface for managing git repos called GitHaven (my personal home install is here: http://git.meyerhome.net:8080/). It has the usual goodies, private/public repos etc. I have some friend's companies that run it in the cloud already. Do you think it would actually make money i.e. Would you be willing to join me in a startup on it (in boston)? Would it pass the y-combinator test?

Re: GitHub major issues - repos have "lost" commits and site is erroring

#68

Post by saurik » Sun, Jan 30, 2011, 4:43 PM UTC

ekidd wrote: post #56

jshen wrote: post #43

Earlier quoted context omitted.

The beauty os a dvcs is that I have a full copy on my machine as well. But seriously, if you use git over ssh there is no sys admin work other than running apt-get upgrade

There's always sysadmin work. 1) You need to keep current on your security patches. 2) You need to upgrade your OS when it reaches end-of-life. 3) You need to make backups, and more importantly, verify that those backups will actually restore. (For git, this is most critical if you have a large team and dozens of repos.) You can ignore a server for years, but eventually you'll get compromised or loose a hard drive. (…

I'd say "most developers" already have a box running some kind of website where they are doing this anyway, but even if you disagree I think you have to be willing to cede "many", especially given how many people who use GitHub are currently doing Ruby on Rails or node.js work. I'd even go so far as to say that those that currently aren't /should/, as the experience being a sysadmin is important when understanding how other sysadmins will react when they see how your software is deployed, which I guess is another topic of discussion that comes up often here (oft filed under the "Debian vs. Ruby" banner).

(Part of me is wondering if many of the more controversial discussions on this site are between people who have sysadmin experience (and considered it valuable) and people who don't.)

Re: GitHub major issues - repos have "lost" commits and site is erroring

#69

Post by saurik » Sun, Jan 30, 2011, 4:49 PM UTC

jrockway wrote: post #35

ericmsimons wrote: post #15

I hope this has nothing to do with the people who hacked sourceforge..

Git has full cryptographic history (and data) authentication, so any changes to the history would be easily detectable. And you can sign known-good commits, so that even if you've never pulled before, you can still verify the part of the history that's been signed. SVN and CVS are missing this key feature, which is why the sf.net hacking is scary.

Git's hashing detects data corruption to an existing user, but to someone who hasn't downloaded the repository yet and is getting it for the first time you really have to remember that SHA1 is currently considered "broken". It is increasingly feasible for someone to generate a collision to an existing file, allowing them to forge a commit. This has been discussed on the git mailing list, for the record, and the response is generally "don't care, we don't claim this to be a security mechanism". This is even a problem if you use signed tags, as you are only signing the result of a broken hash function.

Re: GitHub major issues - repos have "lost" commits and site is erroring

#70

Post by jshen » Sun, Jan 30, 2011, 4:56 PM UTC

ekidd wrote: post #56

jshen wrote: post #43

Earlier quoted context omitted.

The beauty os a dvcs is that I have a full copy on my machine as well. But seriously, if you use git over ssh there is no sys admin work other than running apt-get upgrade

There's always sysadmin work. 1) You need to keep current on your security patches. 2) You need to upgrade your OS when it reaches end-of-life. 3) You need to make backups, and more importantly, verify that those backups will actually restore. (For git, this is most critical if you have a large team and dozens of repos.) You can ignore a server for years, but eventually you'll get compromised or loose a hard drive. (…

If all you're doing on your server is git, then the first two points you make are covered basic apt-get usage. It's super easy.

point 3 isn't that necessary if all you're doing is git over ssh. All the people that have a checked out copy have a backup of your repo. Also, if you're only doing git over ssh then it's not hard to make the box relatively secure. ssh will be hte only open port, and it won't be on the standard port.