[deleted]
Proposed server purchase for GitLab.com
181–190 of 327 posts
Re: Proposed server purchase for GitLab.com
#182Here's a crazy but interesting suggestion: > Backup > ...Even with RAID overhead it should be possible to have 480TB of usable storage (66%). Quoting https://code.facebook.com/posts/1433093613662262/-under-the-... : > Fortunately, with a technique called erasure coding, we can. Reed-Solomon error correction codes are a popular and highly effective method of breaking up data into small pieces and being able to easily…
Any halfway experienced storage engineer is fluent in ECC. You don't need secret sauce from Facebook. That said, because of the first statement, a lot of today's storage solutions will use ECC on the backend if they present you with a logical FS. So you may not (should not?) need to reinvent this wheel. At Facebook's scale this absolutely makes sense, but they're not particularly breaking new ground here. Look at RAI…
I'm still figuring all of this out, thanks for the headsup. I realize Facebook don't have anything particularly interesting, but now I understand just how standard this is, huh.
Thanks again.
Re: Proposed server purchase for GitLab.com
#183Earlier quoted context omitted.
Great points. We'll make sure to wire to separate power feeds that can both handle the entire load. Suggestions in how to calculate this? Taking the maximum rated load seems over the top.
I recently had to do this. The server I was putting up was rated for 3kW. To determine the expected load, I put it under a dummy load that I reasonably considered the maximum for what I would expect on the server (this was a dev machine, so I picked compiling the linux kernel as a benchmark). I ran that until the power stabilized (SuperMicro servers can measure power consumption in hardware and expose this via IPMI -…
I love the idea of killing the server instead of the circuit, thanks!
Re: Proposed server purchase for GitLab.com
#184I'm a cranky old person now, I think this is a crazy approach to take and I would be having a very challenging conversation with the engineer pitching this to me. My underlying assumption is that this is a production service with customers depending on it. 1. Don't fuck with networking. Do you have experience operating same or similar workloads on your super micro sdn? Will the CEO of your super micro VAR pickup his…
If you're going to spend top dollar on Arista/Cisco/EMC/NetApp, you might as well stay in the cloud.
None of the clouds use any of that super-expensive gear, so if you're going for cost savings, you'll need to use the same sort of commodity gear they use.
Gitlab is obviously Linux-savvy and comfortable writing automation, so things like Cumulus Linux and minimal-handholding hardware vendors shouldn't cause them any indigestion.
Re: Proposed server purchase for GitLab.com
#185Earlier quoted context omitted.
Been a softlayer customer for 4 years now. Their network is pretty awesome. When hosted in the same datacenter it's sub-ms response time always. If there is an issue they get right on it. You can even ask them to host the stuff in the same rack to get even better response time.
We are also a SL customer -- 4-figures of hosts with them. We have had networking problems in the past (latency and loss far higher than I would expect to see in a well-provisioned DC) and talked to them about it. It ended up being contention with another customer, it got fixed, and our network performance has been great since. I would encourage you to look at what you can get without trying to do your own colo. You'…
Thanks for the suggestion. Any idea if they can offer 40 Gbps networking?
Re: Proposed server purchase for GitLab.com
#186Earlier quoted context omitted.
By Tiering, I'm talking about moving blocks of data from slower to faster or visa verse. If I have 10TB of 10k IOPS storage and 100TB of 1k IOPS storage in a tiered setup, data that is frequently accessed would reside in the 10k IOPS tier while less frequently accessed data would be in the 1k tier. In this case, the blocks of popular repositories would be stored in SSD, while the blocks of your side project that you…
That is pretty awesome. Should the SSD's for the fast storage be on the OSD nodes that also have the HDD's or should it be separate OSD nodes?
That would be something I would test. I don't know Ceph, so I would be taking a shot in the dark. I would guess it would not make much of a difference as everything is block level. I, personally, would do 1x PCIe SSD for cache, 1x 2/4TB SSD, and 2x 4TB HDD for each storage node.
Edit. If Ceph is smart enough, it would be aware of the tiers present on the node, and would tier blocks on that node. So a Block on node A will stay on node A.
Re: Proposed server purchase for GitLab.com
#187Earlier quoted context omitted.
These are all points a competent engineer would raise :)
I've found that being considered a 'competent engineer' merely means 'never too arrogant to learn'. I didn't know half of the stuff in grandparents' post.
I was attempting to confirm that these are all points are important. Being too arrogant to learn and being in incompetent are two entirely different things.
Re: Proposed server purchase for GitLab.com
#188Earlier quoted context omitted.
- We would continue to use our existing cloud hosted Chef server for now. - We want to use Kubernetes instead of virtualization. - We provisioned 2 spare database servers for that reasons. - Git already compresses files with zlib so I'm not sure we can compress it much further https://git-scm.com/book/uz/v2/Git-Internals-Packfiles - The servers have a separate management port and "Apart from those routers we'll have…
> The servers have a separate management port and "Apart from those routers we'll have a separate router for a 1Gbps management network." It was followed with "For example to make STONITH reliable when there is a lot of traffic on the normal network". That sounded like you want the heartbeats on it, not out of band management.
I probably made a mistake here. I assumed that since the management network was never congested we could run the heartbeats there. But it might be that the server can't address the management port from the operating system.
Re: Proposed server purchase for GitLab.com
#189This undertaking seems like a huge investment in time & overhead for only 64 servers. My guess is that with this move, performance will go up and availability will go down. I can understand the need for performance but if it were my business I would have taken a significantly different approach.
It likely not be 64 servers for long.
Re: Proposed server purchase for GitLab.com
#190If the plan is still to build a huge Ceph-backed filesystem to store your git repos on, you are doomed. Redhat, if you're out there: Now would be a good time to chime in about the limits of Ceph and the reasonable size of a filesystem.
We'll have a Ceph expert review our configuration.