Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Proposed server purchase for GitLab.com

about.gitlab.com

Post reply on HN
Print view View original

51–60 of 327 posts

Re: Proposed server purchase for GitLab.com

#51

Post by BJanecke »

nik736 wrote: post #8

Are you sure about the location? I would go with Frankfurt, Germany. Biggest IX in the world and if you want a "low-latency" solution for all users this is basically the middle of everything. NYC will have a worse connection to Asia and I don't want to begin with India or something. While Frankfurt is basically only 70ms away from NY and around 120ms to the west coast, while even south america should be Just as a sug…

So what about the data privacy in the USA, and patriot act and all that. Also think fin tech where geolocation means more things than just latency.

Re: Proposed server purchase for GitLab.com

#52

Post by rhizome »

compumike wrote: post #42

The raw performance benefits of bare metal vs cloud are incredible, but why does that necessarily mean building & maintaining your own hardware when you can lease (or work out whatever financing you want, but still let the hosting company maintain a lot of the responsibility for HW)? And besides financing, taking on all the HW maint? I'm not sure your needs are so unique as to require custom hardware. You're talking…

To be sure, I've seen benchmarks (can look up later) that found a 10x+ performance improvement over and above the cost savings, leading to I guess a 100x price/performance improvement.

Re: Proposed server purchase for GitLab.com

#53

Post by BJanecke »

compumike wrote: post #42

The raw performance benefits of bare metal vs cloud are incredible, but why does that necessarily mean building & maintaining your own hardware when you can lease (or work out whatever financing you want, but still let the hosting company maintain a lot of the responsibility for HW)? And besides financing, taking on all the HW maint? I'm not sure your needs are so unique as to require custom hardware. You're talking…

Right now that makes sense yes. What about 5 years from now when it doesn't? This way they get to start building a team and culture to support that kind of infrastructure and wean the baby teeth on nice soft furniture.

Re: Proposed server purchase for GitLab.com

#54

Post by olegkikin »

nik736 wrote: post #8

Are you sure about the location? I would go with Frankfurt, Germany. Biggest IX in the world and if you want a "low-latency" solution for all users this is basically the middle of everything. NYC will have a worse connection to Asia and I don't want to begin with India or something. While Frankfurt is basically only 70ms away from NY and around 120ms to the west coast, while even south america should be Just as a sug…

It really depends on where most of their high-paying customers are. Which is, very likely, in the US/Canada and some of EU. So US East Coast makes sense.

Re: Proposed server purchase for GitLab.com

#55

Post by yuhong »

skuhn wrote: post #49

For server hardware, the Supermicro 2U Twins are a reasonable choice, but I prefer their 4U FatTwin chassis. The engineering quality is a little better IMO, and the cost increase isn't too big. Absolutely do not buy their 1U Twin systems, they are hot garbage. The FatTwin chassis has similar density, and can support either 1U half width or 2U half width systems in a particular chassis. Typically I use 1U's for app /…

And if you want to upgrade later, I think you should estimate about ~3 years before 128GB DDR4 LR-DIMMs are cost effective, right?

Re: Proposed server purchase for GitLab.com

#56

Post by dsr_ »

sytse wrote: post #18
octo_t wrote: post #12

It seems based on this post that all the hardware is going to end up in one datacenter. that seems like a big risk if that datacenter ends up having issues. Perhaps look into how to have the nodes split between 2 or 3 different datacenters?

We've considered having part of the servers in a different location. But it is a big risk to see if Ceph can handle the latency. We're also considering using GitLab Geo to have a secondary installation. It seems that data centers can have intermittent network or power issues but they are less likely to go down for days (for example Backblaze is in one DC). At some point we'll likely have multiple datacenters but our…

Do you have a disaster recovery plan that starts with "A meteor has destroyed our primary data center."?

I do; that's my default scenario. If you can survive that, you can survive all sorts of smaller issues like network congestion, data center power problems, grid power problems, and zombie plagues (or flu, which is more likely.)

Re: Proposed server purchase for GitLab.com

#57

Post by fabian2k »

compumike wrote: post #42

The raw performance benefits of bare metal vs cloud are incredible, but why does that necessarily mean building & maintaining your own hardware when you can lease (or work out whatever financing you want, but still let the hosting company maintain a lot of the responsibility for HW)? And besides financing, taking on all the HW maint? I'm not sure your needs are so unique as to require custom hardware. You're talking…

When I looked at how expensive a rented dedicated server was compared to AWS I expected a big difference, but I was still suprised how cheap you can get dedicated servers today. Not worrying about actually owning the hardware is nice, and you still get vastly more hardware for the same price as in the cloud.

There are many advantages to AWS and similar services, but if you can't really take advantage of all the goodies because e.g. you also need to provide a local version of your software (which is the case for Gitlab, as far as I understand), renting dedicated servers is an order of magnitude cheaper.

Re: Proposed server purchase for GitLab.com

#58

Post by BJanecke »

lykron wrote: post #4

If you are looking for performance, do not get the 8TB drives. In my experience, drives above 5TB do not have good response times. I don't have hard numbers, but I built a 10 disk RAID6 array with 5TB disks and 2TB disks and the 2TB disks were a lot more responsive. From my own personal experience, I would go with a PCIe SSD cache/write buffer, and then a primary SSD tier and a HDD tier. Storage, as it seems you guys…

Don't have hard numbers you say?

https://www.backblaze.com/blog/hard-drive-reliability-stats-...

[EDIT]

Oops that's reliability, I can jump the gun sometimes, sorry.

Re: Proposed server purchase for GitLab.com

#59

Post by HolyLampshade »

nik736 wrote: post #8

Are you sure about the location? I would go with Frankfurt, Germany. Biggest IX in the world and if you want a "low-latency" solution for all users this is basically the middle of everything. NYC will have a worse connection to Asia and I don't want to begin with India or something. While Frankfurt is basically only 70ms away from NY and around 120ms to the west coast, while even south america should be Just as a sug…

To that end I'd be cautious of facilities in NJ. If you are dead-set on the US, Ashburn, or even better further inland such as Chicago or Dallas, may be better suitable if you do not plan on having redundancy in another facility.

Frankfurt may be a better option solely on the basis that it is more "environmentally" stable (in terms of events that may disrupt the operation of the facility).

Re: Proposed server purchase for GitLab.com

#60

Post by z3t4 »

Ive done MMO setups that are more simple. You do not need this large cluster. Spread it out with nodes plus backup nodes near customers. Separate customers. Scaling will be as easy as adding more servers one case at a time.
Post reply on HN