Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Proposed server purchase for GitLab.com

about.gitlab.com

Post reply on HN
Print view View original

161–170 of 327 posts

Re: Proposed server purchase for GitLab.com

#161

Post by a_t48 »

tootie wrote: post #112
sytse wrote: post #97

Earlier quoted context omitted.

We looked at providers such as Softlayer but while they guarantee the performance of the servers they typically can't guaranty network latency. Since we're doing this to reduce latency https://about.gitlab.com/2016/11/10/why-choose-bare-metal/ this is essential to us. We'll be glad to look into alternatives that manage the servers and network for us although the argument in https://news.ycombinator.com/item?id=131534…

None of that would convince me to lift a finger. Not unless you put dollar amounts on one vs the other. Version control isn't something I even care about performance.

I do. We had horrible download\merge\resolve times at my place because of some poor choices (storing many copies of large binary files across all branches in perforce) and it made a huge improvement in development time when those issues were resolved. Granted, this was an extreme case, but VC performance is not irrelevant to the cost of software development.

Re: Proposed server purchase for GitLab.com

#162

Post by i336_ »

Here's a crazy but interesting suggestion:

> Backup

> ...Even with RAID overhead it should be possible to have 480TB of usable storage (66%).

Quoting https://code.facebook.com/posts/1433093613662262/-under-the-...:

> Fortunately, with a technique called erasure coding, we can. Reed-Solomon error correction codes are a popular and highly effective method of breaking up data into small pieces and being able to easily detect and correct errors. As an example, if we take a 1 GB file and break it up into 10 chunks of 100 MB each, through Reed-Solomon coding, we can generate an additional set of blocks, say four, that function similar to parity bits. As a result, you can reconstruct the original file using any 10 of those final 14 blocks. So, as long as you store those 14 chunks on different failure domains, you have a statistically high chance of recovering your original data if one of those domains fails.

Facebook didn't release the system they used to do this. I can see two reasons why not to: desire for competitive edge; or the implementation not being a general-purpose solution.

Considering Facebook's general openness, I say get in touch, just in case! It's quite possible that you might be able to figure out something interesting.

I suspect the reason the system wasn't released was due to the latter case - it seems to be technically quite simple and easily achievable for a[ny] company full of algorithms Ph.Ds.

Re: Proposed server purchase for GitLab.com

#163

Post by sytse »

devicenull wrote: post #143

> Each of the two physical network connections will connect to a different top of rack router. We want to get a Software Defined Networking (SDN) compatible router so we have flexibility there. We're considering the 10/40GbE SDN SuperSwitch (SSE-X3648S/SSE-X3648SR) that can switch 1440 Gbps. Noooooo. Do not go for Supermicro for anything where the operating system matters. Their software quality leaves a lot to be de…

Thanks, Cumulus Linux seems to be the way to go.

Re: Proposed server purchase for GitLab.com

#164

Post by darklajid »

sytse wrote: post #24
neom wrote: post #10

Earlier quoted context omitted.

I'm curious to know if you think this work is within the core competency of GitLab. if so, how did you decide it was. If not, how do you realize the investment over time in something that isn't? Is the GitLab CEO here?

GitLab CEO here. Hardware and hosting are certainly not our core competencies. Hence all the questions in the blog post. And I'm sure we made some wrong assumptions on top of that. But it needs to become a core competency, so we're hiring https://about.gitlab.com/jobs/production-engineer/

I love that this question baited your well-known introduction out again. Wasn't there a post at some point that searched for 'GitLab CEO here' on this very site, just for kicks and giggles?

Re: Proposed server purchase for GitLab.com

#165

Post by sytse »

tomschlick wrote: post #125

I didn't see it mentioned but what are your plans for the network strategy. Are you planning to run dual-stack IPv4/IPv6 ? IPv4 only? Internal IPv6 only with NAT64 to the public stuff? Hopefully IPv6 shows up somewhere in the stack. It's sad to see big players not using it yet.

Probably IPv4 on a /24 block but we'll open up a vacancy for a network engineer.

Re: Proposed server purchase for GitLab.com

#166

Post by viraptor »

sytse wrote: post #160
viraptor wrote: post #138

As someone already mentioned "all the other services" are missing (dns, ntp, monitoring, etc.), but also: - Shouldn't there be a puppet / chef / whatever deployment coordinator in there somewhere? - There's no mention of a virtualisation environment. While it's not a hardware issue really, all the extra services mentioned before will not take the whole server and you'll want to collocate some of them. (maybe even som…

- We would continue to use our existing cloud hosted Chef server for now. - We want to use Kubernetes instead of virtualization. - We provisioned 2 spare database servers for that reasons. - Git already compresses files with zlib so I'm not sure we can compress it much further https://git-scm.com/book/uz/v2/Git-Internals-Packfiles - The servers have a separate management port and "Apart from those routers we'll have…

> The servers have a separate management port and "Apart from those routers we'll have a separate router for a 1Gbps management network."

It was followed with "For example to make STONITH reliable when there is a lot of traffic on the normal network". That sounded like you want the heartbeats on it, not out of band management.

Re: Proposed server purchase for GitLab.com

#167

Post by krenoten »

pyrox420 wrote: post #100
sytse wrote: post #97

Earlier quoted context omitted.

We looked at providers such as Softlayer but while they guarantee the performance of the servers they typically can't guaranty network latency. Since we're doing this to reduce latency https://about.gitlab.com/2016/11/10/why-choose-bare-metal/ this is essential to us. We'll be glad to look into alternatives that manage the servers and network for us although the argument in https://news.ycombinator.com/item?id=131534…

Been a softlayer customer for 4 years now. Their network is pretty awesome. When hosted in the same datacenter it's sub-ms response time always. If there is an issue they get right on it. You can even ask them to host the stuff in the same rack to get even better response time.

in the past they've spread servers into different rooms that had interconnects that challenged a team I was on to spend extra time writing compression code. is this better now?

Re: Proposed server purchase for GitLab.com

#168

Post by sulam »

pyrox420 wrote: post #100
sytse wrote: post #97

Earlier quoted context omitted.

We looked at providers such as Softlayer but while they guarantee the performance of the servers they typically can't guaranty network latency. Since we're doing this to reduce latency https://about.gitlab.com/2016/11/10/why-choose-bare-metal/ this is essential to us. We'll be glad to look into alternatives that manage the servers and network for us although the argument in https://news.ycombinator.com/item?id=131534…

Been a softlayer customer for 4 years now. Their network is pretty awesome. When hosted in the same datacenter it's sub-ms response time always. If there is an issue they get right on it. You can even ask them to host the stuff in the same rack to get even better response time.

We are also a SL customer -- 4-figures of hosts with them. We have had networking problems in the past (latency and loss far higher than I would expect to see in a well-provisioned DC) and talked to them about it. It ended up being contention with another customer, it got fixed, and our network performance has been great since.

I would encourage you to look at what you can get without trying to do your own colo. You're not at the scale where you should be thinking about that.

Re: Proposed server purchase for GitLab.com

#169

Post by kijiki »

MichaelGG wrote: post #137
wmf wrote: post #83

I have built out a few racks of Supermicro twins. In general I would suggest hiring your ops people first and then letting them buy what they are comfortable with. C2: The Dell equivalent is C6320. CPU: Calculate the price/performance of the server , not the processor alone. This may lead you towards fewer nodes with 14-core or 18-core CPUs. Disk: I would use 2.5" PMR (there is a different chassis that gives 6x2.5" p…

Can you elaborate on what deficiencies 10GBase-T has in server applications?

Latency, power and cost.

The PHY has to do a lot of forward error correction and filtering, so it adds latency (for the FEC), power (for all the DSP) and cost (for the silicon area to do all of the above).

Re: Proposed server purchase for GitLab.com

#170

Post by sytse »

mwcampbell wrote: post #96
sytse wrote: post #5

Earlier quoted context omitted.

Thanks! The decision to move to metal was because of performance problems https://about.gitlab.com/2016/11/10/why-choose-bare-metal/ It is nice that we'll save on costs but we anticipate a lot of extra complexity that will slow us down. So if it wasn't needed we would have stayed in the cloud. But it is interesting that both our competitors (GitHub.com and BitBucket.org) also moved to metal.

Have you considered hosting with Packet.net? You'd be on bare metal, thus solving your performance problems, but you'd still be renting by the hour as you are now, and you wouldn't have to deal with buying your own hardware and all the complexity that comes with that.

I looked at their site and they talk about bring your own block, anycast, and IPv6. But I can't find any information about networking speeds. What if we end up needing 40 Gbps between the CephFS servers?
Post reply on HN