Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Racking Mac Pros

photos.imgix.com

Post reply on HN
Print view View original

321–330 of 331 posts

Re: Racking Mac Pros

#321

Post by skuhn »

MagerValp wrote: post #298
skuhn wrote: post #71

Earlier quoted context omitted.

(I'm the datacenter manager at imgix, and I wrote this article) 1. Yeah, the OS X graphics pipeline is at the heart of our desire to use Macs in production. It's also pretty sweet to be able to prototype features in Quartz Composer, and use this whole ecosystem of tools that straight up don't exist on Linux. 2. I mentioned this elsewhere already, but it is actually a pretty good value. The chassis itself is not a ter…

BSDPy, AutoNBI, and Imagr provides a bleeding edge OS X deployment solution that runs entirely on Linux. OS images can be generated with AutoDMG, and Munki will keep them configured and updated afterwards. Pop into ##osx-server on freenode if you want to talk to the devs.

Thanks, I was aware of AutoDMB and Munki, but the rest are news to me. We'll check them out.

Re: Racking Mac Pros

#322

Post by skuhn »

dzhiurgis wrote: post #261
rsync wrote: post #109

I have two conflicting responses to what I am seeing here ... First, this is awesome. Just like I want to live in a world where people are paying picodollars for cloud storage[1], I also want to live in a world where a bunch of mac pro cylinders are racked up in a datacenter. Very cool. Second, this is complete silliness. I'm not going to go down the rabbithole of flops per dollar, but there is no way that you can't…

Am I the only to think that sheet metal is quite cheap to bend?

Indeed, it is. Servers are expensive; everything built for a datacenter is expensive. It's very easy to make a passive chassis for far less than conventional datacenter gear prices.

A lot of cost estimates have been thrown around (here and elsewhere). The highest that I've seen is $4000 per unit. That is simply absurd. The initial run of prototypes was far less per unit, and this was a small batch made to iron out the kinks. Economies of scale and design tweaks will drive this down even further.

The chassis design is actually quite elegant from a manufacturing standpoint. That's something that I hope will be made evident by follow-up posts that delve into more technical detail.

Re: Racking Mac Pros

#323

Post by skuhn »

kaolinite wrote: post #89
skuhn wrote: post #71

Earlier quoted context omitted.

(I'm the datacenter manager at imgix, and I wrote this article) 1. Yeah, the OS X graphics pipeline is at the heart of our desire to use Macs in production. It's also pretty sweet to be able to prototype features in Quartz Composer, and use this whole ecosystem of tools that straight up don't exist on Linux. 2. I mentioned this elsewhere already, but it is actually a pretty good value. The chassis itself is not a ter…

Thanks for replying and thanks for the article too - great read with some fantastic photography. Really interesting to hear how you provision servers, had no idea that OS X Server came with tools for that, but it certainly makes sense. I wouldn't have thought Apple would have put much time or thought into creating tools for large deployments, but glad to hear that they have.

Thanks, the photography was done by our lead designer, Miguel. I am super impressed at what he's been able to capture in an environment that can easily come off as utilitarian and sterile.

He has some other work online that you might enjoy, not related to Macs or imgix: http://photos.miggi.me/

Re: Racking Mac Pros

#324

Post by skuhn »

acqq wrote: post #285

It's deep in the comments and I really like the sentence: x0054: "You are fitting triangular shaped computers, wrapped into round cases, into square shaped boxes." And place them horizontally. And without additional fans! And surprisingly, if you read skuhn's answers here, for them it all still has sense, financially. And also surprisingly, Apple says it's OK to use the Mac Pros horizontally: https://support.apple.co…

I wish that I could share some of the internal cost analysis that was a big part of the decision process; I've dropped breadcrumbs here and there, but exposing the whole thing just isn't a possibility.

Physically, the Mac Pro itself is really densely constructed. Even with some empty space inside our Mac Pro chassis, the solution is effectively 1U per 2 GPUs. That's pretty dense, and it hits our power target for the current site design, so going denser would only lead to stranding space ahead of power (which leads to cost inefficiencies).

But, let's consider some hypothetical configs with list prices that I just looked up. Anyone can do this, and these are not reflective of my costs (you can always do better than list). In reality, I would do a lot more digging on the Linux side, but this is a reasonable config that is analogous in performance and fits into my server ecosystem.

I'm excluding costs that would exist either way: the rack itself, CDUs, top-of-rack switch, cabling, and integration labor are all identical or at least very similar. Density is very similar, so there's no appreciable difference in terms of amortized datacenter overhead.

  Mac Pro config (4 systems in a 4U chassis):
    - 4x Mac Pro ($4600)
      - Intel E5-1650 v2
      - 16GB RAM
      - 256GB SSD
      - 2 x D700
    - Our custom chassis

  Capex only: $0.70 per gflop

  Linux config (4 systems in a 4U chassis):
    - SuperMicro F627G2-FT+ ($4900)
      - 4x Intel E5-2643 v2 - 1 CPU each ($1600)
      - 8x 8GB DIMMs - 16GB each ($200)
      - 8x 500GB 7200rpm (RAID1) HDD - 500GB RAID1 boot drive ($300)
      - 8x AMD FirePro S9050 - dual GPU ($1650)

  Capex only: $1.03/gflop
For comparison, I'll give EC2 pricing as well. It's a tad unfair, since we aren't including on-going maintenance and electricity for the Mac or Linux options -- but 3 years of power is also not nearly equal to the cost of a server. EC2's pricing becomes truly atrocious when you consider network costs -- there is simply no comparison between 95th percentile billing and per-byte billing.

  EC2:
    - g2.2xlarge @ 3 year reserved pricing discount ($7410)

  Instance operating cost only: $3.23/gflop
The Linux config for sure offers many more hardware options and greater flexibility -- and it also requires us to rewrite our imaging stack that is working out pretty well for us and our customers.

I firmly believe that we've made a pragmatic and sensible choice for our image rendering platform today. imgix has a number of smart and talented people constantly evaluating and improving our platforms, and I'm confident we will keep making the right decisions in the future (regardless of how nicely the Mac Pro may photograph).

Re: Racking Mac Pros

#325

Post by skuhn »

ansible wrote: post #123
skuhn wrote: post #105

Earlier quoted context omitted.

The Tesla K80 didn't exist when I started this project, but to do some quick math: K80 gflop/s: 8740 2x FirePro D500 gflop/s: 3500 K80 runs about $4900 a card, whereas the entire Mac Pro (list price) is $4000. So it's 2.5x the performance at easily 2x the cost if not more. You're right that there is a cost advantage to going with commodity server hardware, but I don't think it's as great as most people think in this…

2x FirePro D500 gflop/s: 3500 That 3500 gflop/s for the D700? It is instead 2200 for the D500. http://www.amd.com/en-gb/solutions/workstations/d-series K80 runs about $4900 a card, whereas the entire Mac Pro (list price) is $4000. So it's 2.5x the performance at easily 2x the cost if not more. The 6GB VRAM version with the D700 costs another $600 USD each. The K80 has 12GB VRAM per GPU (24GB total per card). If your…

Actually, I realized that we were both wrong on the math.

2200 gflop or 3500 gflop are the specs for just one of the Fire Pro cards. Whoops, I was writing a lot of comments that day.

So a Mac Pro with D700 GPUs has 7000 gflop/s and runs $4600 (list), whereas the Tesla K80 has 8740 gflop/s and runs $4900 or so. Since you still need a whole server to go with the K80, I stand by my thinking that it's not a great deal. We also don't need 12GB of VRAM for our use case, so that's a bit of a waste.

In Nvidia's product line, price/gflop is not at its best in their highest end cards. AWS uses the Nvidia GRID K2, for instance. You're paying a lot for the double precision performance in the Teslas, and imaging doesn't need it.

Re: Racking Mac Pros

#326

Post by skuhn »

anoother wrote: post #162

It would be interesting, once these have been in use for a while, to see some stats on the relative temperatures (+ fan speeds) of each machine within the enclosure. I can imagine the dynamics of 4 machines scavenging air from a single chamber, with an opening on one end, will result in the machines nearer the warm aisle having to work harder to keep cool... I also wonder what kind of ducting could be implemented to…

> I wish I had the chance to work on something like this!

I should have added, for anyone reading: you do! imgix is hiring, and if you don't see a job description that appeals to you, just reach out and let's see. I'm writing ops-type job descriptions right now.

Re: Racking Mac Pros

#327

Post by lunarecho »

gh02t wrote: post #309
x0054 wrote: post #292

Earlier quoted context omitted.

What tools is OS X lacking? From my experience most of the development and server tools are available natively on OS X. It lacks support for containers, but that would be a worthy addition, and I would say worth spending money and time on. The rest is already there for the most part. Developing further their server infrastructure would allow Apple to make a play for the corporate market. Any way, it's a silly argumen…

It's not small server stuff like Apache that they are missing. It's stuff like distributed failover, exotic driver support, SAN, management etc. that they are missing. Big data center stuff, the kind of thing companies like Red Hat make. Those kinds of products are huge investments. Sure Apple might be able to market towards the enterprise, but they simply don't think there is any money to be made. They used to have…

My background:7 Xserves still in production here in K-12 education, 1000+ users in OpenDirectory

In the pipeline:Migrating to the new shiny Mac Pros along with OS X Server

Reasons: Thunderbolt 2 connectivity is amazing and works fine to connect FibreChannel RAIDs. OS X Server: Though it's correct that the GUI got simplified a bit, it's the same server package and complex as it always has been, however easy enough to support. And if configured correctly, a solid workhorse for many scenarios: network accounts for lab use, calendar and contacts server, along with some helper tools it works in heterogene environments fine, supports huge amounts of users in via LDAP..just to name some reasons. for 20 bucks the best server os to support Mac and iOS clients. And because the underlying foundation is UNIX, it's friendly with any networking stuff such as RADIUS for your WP2-Enterprise wi-fi needs..just to name a view.

One thing that is not quite right in the post above: SAN support exists via XSAN.

Re: Racking Mac Pros

#328

Post by prirun »

danielsamuels wrote: post #294
rbanffy wrote: post #256

Earlier quoted context omitted.

> People are having kernel panics and struggling to keep their machines running with current software. Not here. My Mac is at about 11 days of uptime and it's under constant use. At this moment, I can't say it's less reliable then my Linux machines. In this specific case, however, I'd consider ditching the enclosure and ducting cold air through the internal chassis/heatsink. A Macpro is, essentially a heatsink with b…

Is 11 days of uptime supposed to be impressive?

Here's my 10.6.8 Snow Leopard MacBook Pro laptop:

sh-3.2# uptime 11:28 up 117 days, 19:13, 4 users, load averages: 0.91 0.98 0.95

Re: Racking Mac Pros

#329

Post by x0054 »

fragmede wrote: post #303
x0054 wrote: post #271

Earlier quoted context omitted.

At it's core OS X is Unix. In what way would Linux be a better choice? I am not saying that Linux is a worse choice, but for a company that writes an OS as one of it's core businesses, it only makes sense to run that OS in as many places as possible. For one, by running OS X as a server OS they would necessarily spend more time on development and improvement of the OS core. This would pay off in the long run by furth…

> In what way would Linux be a better choice? Well, it's a supported operating system on machines that aren't cylindrical.

Well, so is OS X. It runs on Mac Mini, no :)

Re: Racking Mac Pros

#330

Post by frugalmail »

panglott wrote: post #314
frugalmail wrote: post #253

Earlier quoted context omitted.

>Why pay 5-10X as much to host on AWS? You have no idea what the comparison is, and I don't either. But again, the criticisim is around running a business off of a bunch of Apple "trash cans". >advocating vendor lockin to windows. Linux is no lockin, Windows lockin via software vs. Apple for hardware and software. >"vendor lockin" to that platform Java, Scala,or any other JVM language protects from that, and to a les…

Platform lockin isn't exactly vendor lock-in, but there's a kind of lockin nonetheless. You're going to be dependent to some extent on your platform whatever your platform happens to be.

>Platform lockin isn't exactly vendor lock-in, but there's a kind of lockin nonetheless. You're going to be dependent to some extent on your platform whatever your platform happens to be.

So you making your own chips off of beach sand or something? /s After a certain point you get ridiculous.

JVM and C/C++ (python and other scripting languages to some degree) are the options if you want cross platform environments.

But on a scale of suckiness:

1) Hardware lock in

2) vendor lock in

3) service lock in

4) OS lock in.

5) app server lock in

6) framework lock in

7) Library lock in

8) programming platform lock in

Post reply on HN