Proposed server purchase for GitLab.com

Ignore the part about where the partition is, then.

1. If I look at 2TB vs. 8TB HGST drives, their seek times are 8ms and 9ms respectively. But if you're only using a quarter of the 8TB drive, the drive arm needs to move less than a quarter as much. Won't that save at least 1ms?

2. The 8TB drive has a lot more data per inch, and it's spinning at the same speed. Once a read or write begins, it's going to finish a lot faster.

3. Here's a benchmark putting 8TB ahead in every single way http://hdd.userbenchmark.com/Compare/HGST-Ultrastar-He8-Heli...

The network piece seems kind of incomplete and outdated when compared to what's being discussed in terms of compute and storage. Most of the new networking fabrics being built at this point are 100G, with emerging server connectivity focusing on nx25G for in-rack. Longer-distance 25G is awaiting implementations of FEC but the currently shipping gear can do 3-5M - which is about ideal for top-of-rack. The economics and development cycles of these link-types tends to dictate that 40GE doesn't make sense for a new installation.

The call for "SDN" is incredibly nebulous - to the point of being almost meaningless, IMO. What the big guys tend to be after is a way to control the fabric via standardized API calls - so capability for YANG/NETCONF or some mechanism for direct access to SDE calls. The other thing that's not addressed is how to efficiently get information about the network out of the network. Traditional polled mechanisms (SNMP/RMON, et al) have been shown not to both lack scale and adequate resolution while legacy approaches to push telemetry (sFlow/Netflow) miss the mark in terms of level of detail and compatibility with large-scale data processing needs.

The next point is the selection of topology and the integration with multi-site planning. There's a lot of cool stuff happening in this regard and there seems to still be a pretty big disconnect between what the systems folks seem to understand and what's happening in the network industry, which is a shame because there's probably more opportunity for neat stuff (read: scalability, performance, fault resistance, manageability) than seems to be discussed (at least on HN).

Finally - there's a certain conventional wisdom among the systems and some sections of the programmer crowd that network control planes are just another mostly simple bit of software to be implemented. It's not. It's a hard problem and is the manifest reason why only a handful of organizations have been able to produce software that runs a meaningful percentage of large-scale L3 infrastructure in the world (hint: Arista is a great company but isn't included in this number quite yet). Truly rugged, useful/usefully-featured and performant network code is hard. Making that code work in the context of 30+ years of protocol implementations, morphing standards and a world of bad/clueful actors is REALLY hard. There's an inverse relationship between the amount of money spent on a solution and the amount of specialized expertise you have on staff. A more traditional commercial solution might be more expensive but it also relieves you of the need to keep some relatively rare, likely expensive and almost certainly non-revenue producing skill-sets off-staff.

I'm trying to think of the reason why NFS's latency is tolerable but S3's wouldn't be. (Not that I disagree, you're totally right, but why is this true in principle? Just HTTP being inefficient?)

I would imagine any implementation that used S3 or similar as a backing store would have to heavily rely on an in-memory cache for it (relying on the content-addressable-ness heavily) to avoid re-looking-up things.

I wonder how optimized an object store's protocol would have to be (http2 to compress headers? Protobufs?) before it starts converging on something that has similar latency/overhead to NFS.

We moved 10 racks of servers from New York hosting to custom build small hosting facility in Tallinn, Estonia, and we were able to save about €13K a month on hosting. We use free side cooling to save on electricity. We are here for 3 years now, no major problems so far...

Agreed. Application Architecture is far more important than Cloud vs. Bare Metal. It is just easier and more cost effective to through more bare metal hardware at the problem than it is cloud instances. For some this does make bare metal the better option.

To add to my previous comment though, AWS (and cloud in general) tends to make much more sense if you are utilizing their features and services (Such as Amazon RDS, SQS, etc.), and if you aren't using these services I can absolutely guarantee I can deliver a much lower TCO on bare metal than AWS. (Which is why I offered to consult for them) I see this all the time. Company moves from bare metal to AWS as bare metal is getting expensive, then they quickly find out AWS can't deliver the performance they need without massive scale (because they aren't using a proper salable distributed design and can't afford to re-architect their platform)