Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Racking Mac Pros

photos.imgix.com

Post reply on HN
Print view View original

51–60 of 331 posts

Re: Racking Mac Pros

#51

Post by toomuchtodo »

mozumder wrote: post #23
toomuchtodo wrote: post #14

Earlier quoted context omitted.

In some cases, you have no choice. One project I worked on was where we needed to use proprietary software that only worked on OSX that would take a video, perform waveform analysis on the audio, and the output would be a properly timed closed captioned master with the text having been provided separately. This was of course a small project, and only had a few Mac Minis rack mounted for the task, but I can easily see…

Another is that Macs have their own raw image convertors, and if you're trying to sync photos to the cloud with raw files, and you want to match images with mac users, you're going to need OS X in the cloud. If you don't have OS X in the cloud, then you're going to have to write your own raw image converter, and that means you can't sync with the OS X client native raw converter, complicating the workflow...

You're violating OSX license terms if you attempt to run OSX "in the cloud". That was the first idea I wanted to explore when the project was brought up, but for various legal reasons, we just bought the hardware. (Sidenote: Nothing is more frustrating than sitting in a technical meeting where an attorney has joined in, and explains you can't do something legally even though its the perfect technical solution).

Not to say you can't run OSX virtualized...

Re: Racking Mac Pros

#52

Post by spitfire »

bane wrote: post #29

It's really kind of mind-boggling that Apple makes and sells the Pro, which can be upgraded to a really nice high performance GPU workstation, but then doesn't sell the same hardware in rack mountable forms for clusterable computing. I'm sure they've performed some kind of market analysis for this, but there's enough differences between OSX and Linux solutions that for people who use HPC solutions (a growing market)…

They used to. No one bought them.

Re: Racking Mac Pros

#53

Post by skuhn »

apendleton wrote: post #39
msandford wrote: post #27

Seems like you could have gotten higher density going vertical instead of horizontal. It would have been 50% taller (6U instead of 4U) but it could have held 100% more Mac Pros. A Mac Pro is 9.9 inches tall and 6.6 inches in diameter. 9.9 / 1.75 = 5.65 and 6.6 / 1.75 = 3.77 https://www.apple.com/mac-pro/specs/

I suspect that the sideways orientation is motivated by cooling. The mac pro sucks air in the bottom and shoots it out the top, so if you stack them, the top machine is sucking in the bottom machine's hot air. In this orientation, the whole rack is sucking air in from one exposed side and pushing it out the other exposed side; much easier to engineer around.

You're right, it is entirely about airflow.

Re: Racking Mac Pros

#54

Post by exelius »

mozumder wrote: post #47
exelius wrote: post #33

Earlier quoted context omitted.

Right; which is why I was surprised that OS X stacks up as favorably as it does. But I wouldn't call it a great value since it's still much more expensive than a traditional rack mount setup. From a density perspective it's really not bad at all, which is a testament to how well-engineered the Mac Pro is.

I'd say it's not even as expensive as something like IBM's 4-socket rack servers like the x3850 X6.

Yeah, except IBM's x86 hardware has always been stupid expensive for no good reason other than it's IBM. And didn't they spin off their xSeries server business to Lenovo once the market settled on HP systems that cost half what IBM's did?

Re: Racking Mac Pros

#55

Post by ansible »

exelius wrote: post #5

This is interesting -- they actually manage to get greater density out of this setup than many traditional rack mount systems offer. And to those questioning "Why would you use such expensive systems when commodity hardware is just as fast at half the price?" I would reply that the Mac Pro isn't all that expensive compared to most rack mount servers. If you're talking about a difference of $2000 per server, even acro…

This is interesting -- they actually manage to get greater density out of this setup than many traditional rack mount systems offer.

If they are GPU limited...

A full 4U rack of Mac Pros is 8 AMD Fire GPUs (6GB VRAM each), 256GB main RAM, 48 2.7GHz Xeon cores (using the 12-core option), and 4TB of SSD. 10G Ethernet via Thunderbolt2.

Let's set aside differences in GPU and processor performance; we're just looking at the base stats. All for about $36K USD, not including the rack itself.

An alternative is the SuperMicro 4027GR-TR:

http://www.supermicro.com/products/system/4U/4027/SYS-4027GR...

So, maxed out, you've got 8 Nvidia Tesla K80 cards (dual GPU), 1.5TB RAM, 28 2.6GHz Xeon cores, and a lot of storage (24 hot-swap bays). That's in a 4U rack too.

Call it about $13K USD for the server, and $5K per GPU. Plus a little storage, call it about $56K USD with 10G Ethernet.

The SuperMicro system is designed to be remotely managed. Each GPU has double the VRAM of the AMD Fire ones (12GB vs. 6GB).

I don't know the exact performance figures of the AMD Fire vs. the Kepler GK210, but I'm sure the Fire it isn't nearly as good. And you've got twice as many Nvidia chips on top of that.

At some point its going to get cheaper to re-write the software...

Re: Racking Mac Pros

#56

Post by exelius »

toomuchtodo wrote: post #51
mozumder wrote: post #23

Earlier quoted context omitted.

Another is that Macs have their own raw image convertors, and if you're trying to sync photos to the cloud with raw files, and you want to match images with mac users, you're going to need OS X in the cloud. If you don't have OS X in the cloud, then you're going to have to write your own raw image converter, and that means you can't sync with the OS X client native raw converter, complicating the workflow...

You're violating OSX license terms if you attempt to run OSX "in the cloud". That was the first idea I wanted to explore when the project was brought up, but for various legal reasons, we just bought the hardware. (Sidenote: Nothing is more frustrating than sitting in a technical meeting where an attorney has joined in, and explains you can't do something legally even though its the perfect technical solution). Not t…

You actually can (legally) run OS X "in the cloud", but it has to be on Apple hardware... which kind of defeats the purpose in this case :) But it can be quite useful for development / testing purposes.

Re: Racking Mac Pros

#57

Post by skuhn »

ilzmastr wrote: post #31

Wow, learn something new everyday. I thought everyone who did image processing and cared about performance used NVidia cards for the CUDA libraries. I never knew apple [GPU image libraries]( https://developer.apple.com/library/mac/documentation/Graphi... ) made AMD a competitive choice. It is much more expensive, though a lot less engineering work, than buying some used Tesla's on ebay: http://www.ebay.com/sch/i.html…

Going with a Tesla solution would actually be way way more expensive (when bought new). A Tesla K20 is 3520 gflop/s for $2900 (and then you need a server to put it in). The Mac Pro is 3500gflop/s for $4000.

The Tesla card does have a significant advantage in terms of double precision math, but that isn't the kind of workload we're doing. If we were to go with GPUs on Linux systems, the NVidia GRID card or AMD FirePro server cards are probably a better fit. Or maybe even NVidia Quadro or GTX, although they don't have the proper fan layout and there would be some tears shed over getting the power sockets cabled.

Re: Racking Mac Pros

#58

Post by Sanddancer »

mozumder wrote: post #26
exelius wrote: post #19

Earlier quoted context omitted.

The Mac Pro isn't a great value in the datacenter space. It's a single socket server that's limited to 64 GB of RAM. It's not unusual anymore to throw GPUs in rack mount systems; most of them already have the PCIe bandwidth necessary to support 4 big GPUs so it's often just a matter of getting the right riser cards. Compare a Mac Pro to an HP DL360 that can hold 4 8-core Xeons (32 cores total) and over 200GB of RAM a…

Well, they're fitting 4 sockets and 8 GPUS in the space that's normally used by 4 sockets only. Also, if you're trying to sync raw images between OS X clients and the cloud, then you're going to need OS X servers in the cloud. It'll greatly complicate the clients workflow if they can't use their built in raw converters.

At the scale imgix is going for, and given they're already doing a lot of custom architecture work, something like Supermicro's GPGPU chassis [1] would allow the same server density, plus use GPUs and CPUs that are 1-2 generations ahead of Apple's offerings. Regarding raw images, you don't need OS X servers to do that, just programs that can read the raw formats. That could be a windows box, or an OpenCL-enabled program like darktable [2]. Really the biggest issue here is engineering time for porting the app, and given the costs of the hardware they're using, I'd take a good hard look at how long it would take to port the software; I'd bet that they'd save money after deploying a few boxes.

[1] http://www.supermicro.com/products/system/2u/2028/SYS-2028GR...

[2] https://www.darktable.org/

Re: Racking Mac Pros

#59

Post by derefr »

Apropos to nothing, but when you stuff a bunch of Mac Pros in a box, they begin to look like enlarged vacuum tubes/capacitors. I can almost imagine them being "screwed into" the rack chassis.

Re: Racking Mac Pros

#60

Post by logicallee »

skuhn wrote: post #28
mmastrac wrote: post #2

Building on OSX seems like it must add a ton of complexity to your workflow, despite getting access to some of Apple's GPU-optimized image code. Then again, it's often cheaper to throw silicon at problems than people. If you have in-house expertise in Apple's graphics libraries, that might be cheaper than hiring someone who could write the whole thing to run under a lower-cost Linux solution. Alternatively, OS X migh…

(I'm the datacenter manager at imgix and wrote this post) From a pure hardware perspective, I would love to move this part of the service to Linux systems with GPUs. I spent some time evaluating this before we committed to the Mac Pro solution -- built some prototype hardware and did a cost analysis. It just wasn't the right move, because of the engineering cost for us. OS X's graphics pipeline is really strong, and…

(you should get Apple promotional sponsorship while you're at it. this blog post is great for the mac pro, which is a fantastic piece of hardware.)
Post reply on HN