Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GitLab.com Downtime Postmortem

docs.google.com

Post reply on HN
Print view View original

1–10 of 50 posts

Re: GitLab.com Downtime Postmortem

#2

Post by sytse »

GitLab B.V. CEO here, please let us know if you have any questions (you can also leave a comment or suggestion in the doc if you want). This whole real-time postmortem is something we thought of to contribute back after having downtime. Feel free to let us know what you think of it.

Re: GitLab.com Downtime Postmortem

#3

Post by dvirsky »

sytse wrote: post #2

GitLab B.V. CEO here, please let us know if you have any questions (you can also leave a comment or suggestion in the doc if you want). This whole real-time postmortem is something we thought of to contribute back after having downtime. Feel free to let us know what you think of it.

I haven't read it but I like the idea in general. One question: aren't you concerned about accidentally exposing security critical info over the doc?

Re: GitLab.com Downtime Postmortem

#4

Post by LeonM »

sytse wrote: post #2

GitLab B.V. CEO here, please let us know if you have any questions (you can also leave a comment or suggestion in the doc if you want). This whole real-time postmortem is something we thought of to contribute back after having downtime. Feel free to let us know what you think of it.

It's only after I saw the "B.V." I realized Gitlab is a Dutch company, cool! :)

Just one question, I saw a lot of logs being copy-pasted, aren't you concerned about any security sensitive data leaking out?

Re: GitLab.com Downtime Postmortem

#5

Post by sytse »

dvirsky wrote: post #3
sytse wrote: post #2

GitLab B.V. CEO here, please let us know if you have any questions (you can also leave a comment or suggestion in the doc if you want). This whole real-time postmortem is something we thought of to contribute back after having downtime. Feel free to let us know what you think of it.

I haven't read it but I like the idea in general. One question: aren't you concerned about accidentally exposing security critical info over the doc?

Glad you like the idea. We're worried about exposing sensitive information. In general we're pretty careful about credentials so that will be OK and our setup is not a secret (a lot is open source and we share the High Availability details with standard subscribers). But it could happen that we share user information such as projects names in the log-files by accident. We think that there are also benefits of working out in the open and thing that the trade-off is worth it. We hope that people let us know if they see anything sensitive in the doc so we can quickly remove it.

Re: GitLab.com Downtime Postmortem

#6

Post by sytse »

LeonM wrote: post #4
sytse wrote: post #2

GitLab B.V. CEO here, please let us know if you have any questions (you can also leave a comment or suggestion in the doc if you want). This whole real-time postmortem is something we thought of to contribute back after having downtime. Feel free to let us know what you think of it.

It's only after I saw the "B.V." I realized Gitlab is a Dutch company, cool! :) Just one question, I saw a lot of logs being copy-pasted, aren't you concerned about any security sensitive data leaking out?

Thanks! We are based in the Netherlands but we're mostly a remote company https://about.gitlab.com/2014/07/03/how-gitlab-works-remotel...

We're worried about sensitive data, see my other answer https://news.ycombinator.com/item?id=8003800

Re: GitLab.com Downtime Postmortem

#7

Post by dvirsky »

sytse wrote: post #5
dvirsky wrote: post #3

Earlier quoted context omitted.

I haven't read it but I like the idea in general. One question: aren't you concerned about accidentally exposing security critical info over the doc?

Glad you like the idea. We're worried about exposing sensitive information. In general we're pretty careful about credentials so that will be OK and our setup is not a secret (a lot is open source and we share the High Availability details with standard subscribers). But it could happen that we share user information such as projects names in the log-files by accident. We think that there are also benefits of working…

Thanks, I've now read it. Very nice. I guess having your architecture out in the open is liberating in that sense.

So now two specific questions: 1. So why did you have a big read spike? And 2. are you going to share the TODOs as well? :)

BTW We're using gitlab internally (at EverythingMe) and are very happy about it, especially the flow of fixes and features implemented.

Re: GitLab.com Downtime Postmortem

#8

Post by sytse »

dvirsky wrote: post #7
sytse wrote: post #5

Earlier quoted context omitted.

Glad you like the idea. We're worried about exposing sensitive information. In general we're pretty careful about credentials so that will be OK and our setup is not a secret (a lot is open source and we share the High Availability details with standard subscribers). But it could happen that we share user information such as projects names in the log-files by accident. We think that there are also benefits of working…

Thanks, I've now read it. Very nice. I guess having your architecture out in the open is liberating in that sense. So now two specific questions: 1. So why did you have a big read spike? And 2. are you going to share the TODOs as well? :) BTW We're using gitlab internally (at EverythingMe) and are very happy about it, especially the flow of fixes and features implemented.

1. We don't know (and the read spike might be an effect rather than the cause)

2. Yes, most of the things after the hashrockets (=>) currently in the doc are TODO's

BTW Awesome to hear that you are happy users of GitLab.

Re: GitLab.com Downtime Postmortem

#9

Post by dvirsky »

sytse wrote: post #8
dvirsky wrote: post #7

Earlier quoted context omitted.

Thanks, I've now read it. Very nice. I guess having your architecture out in the open is liberating in that sense. So now two specific questions: 1. So why did you have a big read spike? And 2. are you going to share the TODOs as well? :) BTW We're using gitlab internally (at EverythingMe) and are very happy about it, especially the flow of fixes and features implemented.

1. We don't know (and the read spike might be an effect rather than the cause) 2. Yes, most of the things after the hashrockets (=>) currently in the doc are TODO's BTW Awesome to hear that you are happy users of GitLab.

I bet it's an effect rather than a cause. We had some server incident recently that started with a huge write spike on some machines. Turned out it was the nginx error log and the problem itself was something else.

Re: GitLab.com Downtime Postmortem

#10

Post by sytse »

sytse wrote: post #8
dvirsky wrote: post #7

Earlier quoted context omitted.

Thanks, I've now read it. Very nice. I guess having your architecture out in the open is liberating in that sense. So now two specific questions: 1. So why did you have a big read spike? And 2. are you going to share the TODOs as well? :) BTW We're using gitlab internally (at EverythingMe) and are very happy about it, especially the flow of fixes and features implemented.

1. We don't know (and the read spike might be an effect rather than the cause) 2. Yes, most of the things after the hashrockets (=>) currently in the doc are TODO's BTW Awesome to hear that you are happy users of GitLab.

Right now it looks like 1 was caused by an extremely large repo (18 GB) being pushed. Only 0.1% of repos are bigger than 1GB. We're still investigating.
Post reply on HN