Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Racking Mac Pros

photos.imgix.com

Post reply on HN
Print view View original

121–130 of 331 posts

Re: Racking Mac Pros

#121

Post by pronoiac »

toomuchtodo wrote: post #14
mmastrac wrote: post #2

Building on OSX seems like it must add a ton of complexity to your workflow, despite getting access to some of Apple's GPU-optimized image code. Then again, it's often cheaper to throw silicon at problems than people. If you have in-house expertise in Apple's graphics libraries, that might be cheaper than hiring someone who could write the whole thing to run under a lower-cost Linux solution. Alternatively, OS X migh…

In some cases, you have no choice. One project I worked on was where we needed to use proprietary software that only worked on OSX that would take a video, perform waveform analysis on the audio, and the output would be a properly timed closed captioned master with the text having been provided separately. This was of course a small project, and only had a few Mac Minis rack mounted for the task, but I can easily see…

Hey, I'd like to know more about your speech-to-text project, if you're willing to share.

Re: Racking Mac Pros

#122

Post by sajal83 »

skuhn wrote: post #71
kaolinite wrote: post #13

Three possible reasons I can think of for doing this over using PCs or Linux servers: 1. Using the same operating system as the developers of the software, plus access to Apple's fantastic imaging libraries. 2. The Mac Pro, whilst expensive, is good value for money. The dual graphics cards inside it are not cheap at all. As servers with GPUs are fairly niche, this might actually be a cheaper solution. 3. The form fac…

(I'm the datacenter manager at imgix, and I wrote this article) 1. Yeah, the OS X graphics pipeline is at the heart of our desire to use Macs in production. It's also pretty sweet to be able to prototype features in Quartz Composer, and use this whole ecosystem of tools that straight up don't exist on Linux. 2. I mentioned this elsewhere already, but it is actually a pretty good value. The chassis itself is not a ter…

Why not unassemble the cylinders and re-assemble into rectangle chasis? Im sure that would give you a more dense layout.. Sure it would void warranty and resale value.. but do you really care?

Re: Racking Mac Pros

#123

Post by ansible »

skuhn wrote: post #105
ansible wrote: post #55

Earlier quoted context omitted.

This is interesting -- they actually manage to get greater density out of this setup than many traditional rack mount systems offer. If they are GPU limited... A full 4U rack of Mac Pros is 8 AMD Fire GPUs (6GB VRAM each), 256GB main RAM, 48 2.7GHz Xeon cores (using the 12-core option), and 4TB of SSD. 10G Ethernet via Thunderbolt2. Let's set aside differences in GPU and processor performance; we're just looking at t…

The Tesla K80 didn't exist when I started this project, but to do some quick math: K80 gflop/s: 8740 2x FirePro D500 gflop/s: 3500 K80 runs about $4900 a card, whereas the entire Mac Pro (list price) is $4000. So it's 2.5x the performance at easily 2x the cost if not more. You're right that there is a cost advantage to going with commodity server hardware, but I don't think it's as great as most people think in this…

2x FirePro D500 gflop/s: 3500

That 3500 gflop/s for the D700? It is instead 2200 for the D500.

http://www.amd.com/en-gb/solutions/workstations/d-series

K80 runs about $4900 a card, whereas the entire Mac Pro (list price) is $4000. So it's 2.5x the performance at easily 2x the cost if not more.

The 6GB VRAM version with the D700 costs another $600 USD each.

The K80 has 12GB VRAM per GPU (24GB total per card).

If your code can use the additional memory, that is a huge difference.

Anyway, 3500 gflop/s times 8 is 28 tflop/s for the Mac Pros.

With 8 K80s, you're at 70 tflop/s. Single precision. So that's double the raw performance, and double the memory. Actual performance for a given workload? I wouldn't care to say.

I'd be concerned about thermal issues too. I wouldn't be surprised that the Mac Pro gets throttled after a while when running it hard. The kind of server you can put the K80 in usually has additional (server-grade) cooling.

I'm not disrespecting you guys, if you've got a solution that works, and makes you money, more power to you!

But I stand by my claim that at some point, it will be cheaper to rewrite the software for the render pipeline. Not this year I guess, and who knows, maybe not next year either.

Re: Racking Mac Pros

#124

Post by tinco »

terhechte wrote: post #114
rsync wrote: post #109

I have two conflicting responses to what I am seeing here ... First, this is awesome. Just like I want to live in a world where people are paying picodollars for cloud storage[1], I also want to live in a world where a bunch of mac pro cylinders are racked up in a datacenter. Very cool. Second, this is complete silliness. I'm not going to go down the rabbithole of flops per dollar, but there is no way that you can't…

Sure, you can build such a hackintosh, but you're behind on OSX updates, you can't update automatically (it may break some of the custom hacks required), you probably can't use auto update. Also, some of the drivers may have additional bugs due to the ever-so-slight hardware change. In the end, your system may freeze more often, or display weird behaviour. All that apart from the EULA issue that was already mentioned…

terhechte is right about this. To clarify: the reason you can't update automatically is although most hardware is supported just fine, there are many minor adjustments needed, mostly to text files but sometimes to binaries as well to make OSX recognize your hardware. Every time an OSX update comes out there's updates to drivers that overwrite your modified files.

An example of the sort of hack I'm talking about would be a graphics driver that says it's for the NVidia model E532D. Your graphics card is an E532E. You looked on the internet, and you found out they are exactly identical except for branding, so you dive in the driver and simply flip a bit to make OSX recognize it.

Re: Racking Mac Pros

#125

Post by to3m »

jallmann wrote: post #78

This seems risky from a business perspective: it's voluntary vendor lock-in. What if Apple decides to change the Mac Pro form factor for the next iteration? Then you have to retool and are left with a bunch of incompatible chassis. What if Apple stagnates with hardware upgrades? You'd be stuck running obsolete hardware. What if Apple discontinues the entire Mac Pro line? Not to mention the price premium of Apple hard…

This was pretty much my thought as well. The results of this cost-benefit analysis make me raise my eyebrow, and, same as you, I can only assume that OS X permeates the infrastructure from top to bottom, to an extent that makes pulling it out too painful to even contemplate. Image processing shaders of this type aren't that hard to write.

If you're worried about them not matching some piece of client software exactly (Quartz Composer, Photoshop, etc.), you still have options. And those options - e.g., webapp for previewing/something else/etc. - you'll probably want anyway, for the benefit of designers that don't run OS X.

(The filtering aspect of the system I find a little surprising anyway - the idea of an image-focussed client-aware DPI-aware CDN makes sense to me (and I like it!), but something that does your Photoshop filters in the cloud sounds less compelling. I would have expected people to prefer to do that locally, and upload the result. But... while I've worked with many artists and designers, I'm not one myself. So maybe they'll go for that. And/or maybe a lot of their customers take advantage of the fact that the processing appears to be free. And I'm prepared to contemplate the possibility, however unlikely, that they might know their customer base better than I do.)

Re: Racking Mac Pros

#126

Post by JonathonW »

sajal83 wrote: post #122
skuhn wrote: post #71

Earlier quoted context omitted.

(I'm the datacenter manager at imgix, and I wrote this article) 1. Yeah, the OS X graphics pipeline is at the heart of our desire to use Macs in production. It's also pretty sweet to be able to prototype features in Quartz Composer, and use this whole ecosystem of tools that straight up don't exist on Linux. 2. I mentioned this elsewhere already, but it is actually a pretty good value. The chassis itself is not a ter…

Why not unassemble the cylinders and re-assemble into rectangle chasis? Im sure that would give you a more dense layout.. Sure it would void warranty and resale value.. but do you really care?

The whole machine's custom built to fit inside the cylindrical case... the best you could do would be to take the outer case off, and then you've just got a slightly smaller cylinder.

Electrically, everything's built around a round "central" PCB using a custom interconnect. You're not going to be able to reassemble the thing into a rectangle and still get a functioning machine (not without tons of custom design work, at least).

See https://www.ifixit.com/Teardown/Mac+Pro+Late+2013+Teardown/2...

Re: Racking Mac Pros

#127

Post by MrBuddyCasino »

tinco wrote: post #113
rsync wrote: post #109

I have two conflicting responses to what I am seeing here ... First, this is awesome. Just like I want to live in a world where people are paying picodollars for cloud storage[1], I also want to live in a world where a bunch of mac pro cylinders are racked up in a datacenter. Very cool. Second, this is complete silliness. I'm not going to go down the rabbithole of flops per dollar, but there is no way that you can't…

Obviously, but running OSX on non-Apple hardware is a violation of its EULA. I have contacted a lawyer for this (I wanted to run Hackintosh in the office), the language is very clear. The author of the software has the full power to license its use to you with any restrictions they find necessary no matter how ridiculous. If Apple only sells you the license if you promise not to run it on a thursday, you'll be in vio…

Depends on where you live. At least in Germany (and I think the whole EU), EULAs are meaningless.

Re: Racking Mac Pros

#128

Post by rylee »

skuhn wrote: post #71
kaolinite wrote: post #13

Three possible reasons I can think of for doing this over using PCs or Linux servers: 1. Using the same operating system as the developers of the software, plus access to Apple's fantastic imaging libraries. 2. The Mac Pro, whilst expensive, is good value for money. The dual graphics cards inside it are not cheap at all. As servers with GPUs are fairly niche, this might actually be a cheaper solution. 3. The form fac…

(I'm the datacenter manager at imgix, and I wrote this article) 1. Yeah, the OS X graphics pipeline is at the heart of our desire to use Macs in production. It's also pretty sweet to be able to prototype features in Quartz Composer, and use this whole ecosystem of tools that straight up don't exist on Linux. 2. I mentioned this elsewhere already, but it is actually a pretty good value. The chassis itself is not a ter…

What's so good about this OS X graphics pipeline that isn't on anything else? I'm now super curious.

Re: Racking Mac Pros

#129

Post by listic »

zurn wrote: post #115
skuhn wrote: post #71

Earlier quoted context omitted.

(I'm the datacenter manager at imgix, and I wrote this article) 1. Yeah, the OS X graphics pipeline is at the heart of our desire to use Macs in production. It's also pretty sweet to be able to prototype features in Quartz Composer, and use this whole ecosystem of tools that straight up don't exist on Linux. 2. I mentioned this elsewhere already, but it is actually a pretty good value. The chassis itself is not a ter…

> The power draw related to cooling for this rack versus a rack of conventional servers is about 1-5/th to 1/10th as much. Do you mean your Mac Pros dissipate 1/5 to 1/10th as much heat as other x86 server hardware, or is there there some other factor in play that makes your AC 5-10x more power efficient?

I understand "related to cooling" as Mac Pro's cooling in this setup is 5-10x more efficient.

Re: Racking Mac Pros

#130

Post by chiph »

skuhn wrote: post #107
chiph wrote: post #92

Earlier quoted context omitted.

What's the noise level like with these machines? The typical pizza-box servers aren't exactly quiet.

They're pretty much silent relative to datacenter stuff. One of the goals of the next revision is to have LED power indicators (maybe plugged in to the front USB ports) or LCD panels built into the front of the chassis. Right now you actually can't tell that the rack is powered on unless you walk to the hot aisle and look at the power readouts, it's that quiet.

Is fan failure reported through management APIs?
Post reply on HN