111–120 of 327 posts
Just a few quick notes. I've experience running ~300TB of usable Ceph storage. Stay away from the 8TB drives. Performance and recovery will both suck. 4TB drives still give the best cost per GB. Why are you using fat twins? Honestly, what does that buy you? You need more spindles, and fewer cores and memory. With your current configuration, what are you getting per rack unit? Consider a 2028u based system. 30 of thos…
The raw performance benefits of bare metal vs cloud are incredible, but why does that necessarily mean building & maintaining your own hardware when you can lease (or work out whatever financing you want, but still let the hosting company maintain a lot of the responsibility for HW)? And besides financing, taking on all the HW maint? I'm not sure your needs are so unique as to require custom hardware. You're talking…
We looked at providers such as Softlayer but while they guarantee the performance of the servers they typically can't guaranty network latency. Since we're doing this to reduce latency https://about.gitlab.com/2016/11/10/why-choose-bare-metal/ this is essential to us. We'll be glad to look into alternatives that manage the servers and network for us although the argument in https://news.ycombinator.com/item?id=131534…
Just a few quick notes. I've experience running ~300TB of usable Ceph storage. Stay away from the 8TB drives. Performance and recovery will both suck. 4TB drives still give the best cost per GB. Why are you using fat twins? Honestly, what does that buy you? You need more spindles, and fewer cores and memory. With your current configuration, what are you getting per rack unit? Consider a 2028u based system. 30 of thos…
We're considering the fat twins so we get both a lot of CPU and some disk. GitLab.com is pretty CPU heavy because of ruby and the CI runners that we might transfer in the future. So we wanted the maximum CPU per U.
The 2028u has 2.5" drives. For that I only see 2TB drives on http://www.supermicro.com/support/resources/HDD.cfm for the SS2028U-TR4+. How do you suggest getting to 4TB?
My underlying assumption is that this is a production service with customers depending on it.
1. Don't fuck with networking. Do you have experience operating same or similar workloads on your super micro sdn? Will the CEO of your super micro VAR pickup his phone at 2AM when you call?
My advice: Get a quote from Arista.
2. Don't fuck with storage.
32 file servers for 96TB? Same question as with networking re:ceph. What are your failure domains? How much does it cost to maintain the FTEs who can run this thing?
3. What's the service SLA on the servers? Historically, supermicro VARs have been challenged with that.
If I were building this solution, I'd want to understand what the ROI of this science project is as compared to the current cloud solution and a converged offering like Cisco/NetApp, HP/3Par or something like Nutanix. You're probably saving like 20-25% on hardware.
This sounds to me like pinching pennies on procurement and picking up tens of dollars of cost on the labor side. If you're accustomed to SLAs from AWS, this will be a rude awakening.
If you're committed to having a robust architecture (this may not be financially viable immediately) you should study the mistakes that Github have made, e.g. https://news.ycombinator.com/item?id=11029898 Geo-redundancy seems like a luxury, until your entire site comes down due to a datacenter-level outage. (E.g. the power goes down, or someone cuts the internet lines when doing construction work on the street outsid…
4 x 208v 30A circuits gives a total of 120A -- of which they can only be used at 80% capacity, so that gives you 96A usable -- without redundancy.
My initial feelings (eyeballing it) are you should be looking for 3 full racks, 2x208V@30A in each.
As a juniper shop, we implemented the 2xQFX 5100 48T (in virtual chassis) + ex4300 for remote access per rack. This will be a decision based on your local expertise though.
I also looked hard into the twin boxes -- but power to the rack in the end ruled the day, and it made not much of a difference to use the 1U boxes.
Don't forget about out of band (serial for the switching gear). We've been using OpenGear.com for this stuff with 4G-LTE builtin.
VPN access to access console devices?
Any site-to-site VPN needs?
Also, I would consider not pre-purchasing N times your required horsepower/disk if you can avoid it, but rather add in pre-planned yearly, or biannually stages.
As the CEO of a cloud hosting & server management company -- I have much more to say about this if you would like to chat via phone or email anytime.
Earlier quoted context omitted.
Right now that makes sense yes. What about 5 years from now when it doesn't? This way they get to start building a team and culture to support that kind of infrastructure and wean the baby teeth on nice soft furniture.
I wonder if they could go with a hosting provider, hire a sysadmin, then second them to the hosting provider to learn how it's done, in return for a credit against their costs. In fact, i wonder if that's a business model for hosting providers. Kind of a sysadmin incubator.
It'd be interesting to see more application-level solutions to scale rather than just adding hardware. Like extending git so objects can be fetched via a remote object-store rather than dealing with a locally mounted POSIX file system. This would allow you to use native cloud object stores and might simplify your latency requirements to the point where you could consider staying in the cloud. This definitely increase…
Z2 (no, Z is not on the list you have) You need at least two nodes that do DNS, DHCP, NTP, and other miscellaneous services that you absolutely want to have but do not seem to have mentioned. You want them to be permanently assigned, so that you never have to search for them, and you want them to fail over each service as needed, preferably both operating at once. Three nodes would be better. Consider doing some basi…
Absolutely agree. Typically I deploy about 4-6 "tools" hosts in a site. From a computational standpoint, you could make do with fewer hosts, but there are some things that I prefer to separate out: 1&2. Authoritative DNS (internal) 1&2. NTP 1&2. Caching DNS resolver 3&4. Outbound HTTP proxy (if necessary) 3&4. PXE / installer / dhcp 3&4. Local software mirror (apt / yum / etc.) 5&6. SSH jump hosts. Or something like…
Earlier quoted context omitted.
So what about the data privacy in the USA, and patriot act and all that. Also think fin tech where geolocation means more things than just latency.
Why does latency matter for fin tech for a repository / source control?
Some financial institutions require all sensitive* data to be stored/hosted in the same country/state(archaic yes).
It's real hard to actually define sensitive data but the IP* in some code a quant wrote can totally be considered a trade secret by a non technical person Please don't get me started on how stupid I think it is that people consider code to be IP