Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Racking Mac Pros

photos.imgix.com

Post reply on HN
Print view View original

241–250 of 331 posts

Re: Racking Mac Pros

#241

Post by k2enemy »

rsync wrote: post #109

I have two conflicting responses to what I am seeing here ... First, this is awesome. Just like I want to live in a world where people are paying picodollars for cloud storage[1], I also want to live in a world where a bunch of mac pro cylinders are racked up in a datacenter. Very cool. Second, this is complete silliness. I'm not going to go down the rabbithole of flops per dollar, but there is no way that you can't…

I had the exact same thoughts. On one hand I love the idea of racking mac pros and having drawers full of Wera tools to work with, but on the other hand, it just seems like the kind of silly expenditures that come with too much funding.

Re: Racking Mac Pros

#242

Post by x0054 »

skuhn wrote: post #158
rsync wrote: post #109

I have two conflicting responses to what I am seeing here ... First, this is awesome. Just like I want to live in a world where people are paying picodollars for cloud storage[1], I also want to live in a world where a bunch of mac pro cylinders are racked up in a datacenter. Very cool. Second, this is complete silliness. I'm not going to go down the rabbithole of flops per dollar, but there is no way that you can't…

(I'm the datacenter manager at imgix, and I wrote the article) I've alluded to this elsewhere, but the math doesn't add up to your gut reaction. It's cheaper, but not by a significant enough margin relative to the engineering costs, to go with commodity servers and GPUs. Building things out of sheet metal is actually easier than migrating to Linux, for one big reason: we can pay someone else to do it, because it isn'…

It strikes me that OS X is a solid server platform. Linux is always a more flexible choice, but if OS X works for you, it works for you. That being said, is sticking cylindrical mac pros on their side into square racks really the best solution.

There are 2 problems I see with this design:

1: You are placing the Mac Pros on their side, which may lead to premature bearing failure on the main cooling fan. Apple designed the cooling fan to be as silent as possible, which means that they optimized the bearing and the fan to work in vertical orientation. Bearings designed for thrust (vertical) orientation may not work so well if placed horizontally for a long time.

2: You are fitting triangular shaped computers, wrapped into round cases, into square shaped boxes, resulting in significant loss of space density.

Considering that Apple is a huge company that owns huge data centers, combined with the fact that it would be simply stupid for a company who makes their own OS to run anything but that OS, and combined with the above mentioned problems with using Mac Pros as server "logs" (because you cant call them blades), I would assume that Apple has internally OSX servers designed in the traditional blade configuration.

They may not sell or advertise them, but they MUST have them. Given that you guys are buying a ton of hardware, and are located nearby, and would be actively promoting running Apple hardware, wouldn't it be wise to at least approach Apple and see if they would be kind enough to sell you some of those blade form factor servers they simply must have.

I may be completely wrong here, but apple did brag about how Swift is the new language thats so flexible that you can make a Mobile app in it, or a Desktop app, or even a full blown social network. If that's the case, they must have some plans for the server market? No?

Any way, in the end it's a cool design, but I would seriously consider at least stacking the Mac Pros vertically to avoid fan issues. You can actually get a tighter form factor that way as well, unless space is not the issue. And if it's not, then hell, what's wrong with just placing a bunch of Pros on an Ikea shelf in a well air-conditioned room :)

Re: Racking Mac Pros

#243

Post by skuhn »

k2enemy wrote: post #241
rsync wrote: post #109

I have two conflicting responses to what I am seeing here ... First, this is awesome. Just like I want to live in a world where people are paying picodollars for cloud storage[1], I also want to live in a world where a bunch of mac pro cylinders are racked up in a datacenter. Very cool. Second, this is complete silliness. I'm not going to go down the rabbithole of flops per dollar, but there is no way that you can't…

I had the exact same thoughts. On one hand I love the idea of racking mac pros and having drawers full of Wera tools to work with, but on the other hand, it just seems like the kind of silly expenditures that come with too much funding.

I've answered the concern about Wera hand tools before, but to reiterate my position: a $6 screwdriver doesn't matter if it helps you do your job.

Not having a screwdriver when you need it in a pinch is penny wise and pound foolish. At best you're now out 30 minutes while you drive to Home Depot, potentially during some sort of catastrophe. At worst maybe you simply cannot do the task that you need to do, because it's 2am and you're in Frankfurt. I've worked in a lot of datacenters that didn't stock basic tools to perform tasks, and frankly it sucked.

I keep a log of all of the purchases I made for the current datacenter build. Non-server / non-structural expenses account for less than $3000, which is less than the cost of a single server. This includes storage bins, carts, shelves, workbenches, chairs, supplies and tools.

Re: Racking Mac Pros

#244

Post by [deleted] »

exelius wrote: post #30
robbyt wrote: post #24

Earlier quoted context omitted.

OS X does not have NUMA. It has some nice libraries for multi-threading, but that doesn't really matter that much when you're saturating your memory bus because the CPUs are doing too many cross-zone memory requests.

NUMA is irrelevant anyway because there are currently no multi-socket OS X machines. Multiple cores on the same package share a memory controller.

[deleted]

Re: Racking Mac Pros

#245

Post by MCRed »

blkhp19 wrote: post #217
dinasaurusrex wrote: post #142

Earlier quoted context omitted.

(full disclosure: I work at imgix) Uploading pre-edited images takes time/resources, and in general a lot of our customers rely on us to do all of their image processing so that they don't have to. Additionally, creating edited versions of images in advance presents two problems: 1) Any future site redesigns or edits must now be applied en masse to the existing images or risk older images not complying with the new s…

Hackintoshes would make a lot more sense.

No, they wouldn't.

Look at the current state of Mackintoshes. People are having kernel panics and struggling to keep their machines running with current software. OS X moves pretty fast, possibly faster than Linux, and Apple builds it to support Mac hardware.... the teams who are porting hackintosh code have to support a lot more hardware variety and they have less resources than, say, linux.

Running mackintoshes in production makes no sense.

And I challenge the claim that you would save money.

Looking just at off the shelf costs of low end hardware does not tell you the TCO of serious machines that need to be running all the time.

To get comparable hardware quality to Apple products you have to spend more, generally, when going with "commodity" hardware.

The idea that Apple is expensive is a myth, born of two things-- people perpetuating it since 1980 (yes, 35 years this myth has been spread), with the vested interest of rationalizing their dislike of Apple... and the fact that Apple doesn't compete at the very low end.

In production, TCO is much more about reliability and other things than initial hardware cost.

Re: Racking Mac Pros

#246

Post by skuhn »

x0054 wrote: post #242
skuhn wrote: post #158

Earlier quoted context omitted.

(I'm the datacenter manager at imgix, and I wrote the article) I've alluded to this elsewhere, but the math doesn't add up to your gut reaction. It's cheaper, but not by a significant enough margin relative to the engineering costs, to go with commodity servers and GPUs. Building things out of sheet metal is actually easier than migrating to Linux, for one big reason: we can pay someone else to do it, because it isn'…

It strikes me that OS X is a solid server platform. Linux is always a more flexible choice, but if OS X works for you, it works for you. That being said, is sticking cylindrical mac pros on their side into square racks really the best solution. There are 2 problems I see with this design: 1: You are placing the Mac Pros on their side, which may lead to premature bearing failure on the main cooling fan. Apple designed…

1. That's certainly a possibility, and one that we won't really have hard numbers on for some time to come. However, the fan is about a $60 part, so provided that we don't have coordinated, catastrophic failures and that they live for at least a year, we're doing alright. Do note that Apple specifically says that the Mac Pro may be operated on its side. https://support.apple.com/en-us/HT201379

2. True, but 1U per server is not bad density by any stretch. For my app servers, they effectively occupy 0.5U; database and storage effectively occupy 1U. So this puts the Mac Pros on par with the larger server class. Were we to deploy renderers in conventional server chassis, a similar system would occupy at least an effective 1U if not a full 2U.

What Apple does internally is, of course, shrouded in mystery. I know some people there, and we talk to people when we can, but they just aren't the kind of company that is going to tell you how they make the sausage.

From what I've heard and my sense from speaking with them over the years, they do not use OS X in production. They used to use Darwin and Solaris, and now almost exclusively use Linux (presumably Solaris is still around to run Oracle). They did used to use Xserves internally, but even at their scale it isn't worth building them just for their own use.

Re: Racking Mac Pros

#247

Post by MCRed »

jallmann wrote: post #78

This seems risky from a business perspective: it's voluntary vendor lock-in. What if Apple decides to change the Mac Pro form factor for the next iteration? Then you have to retool and are left with a bunch of incompatible chassis. What if Apple stagnates with hardware upgrades? You'd be stuck running obsolete hardware. What if Apple discontinues the entire Mac Pro line? Not to mention the price premium of Apple hard…

Why pay 5-10X as much to host on AWS? It's not for free.

Hosts have a really nice markup, compared to hosting yourself. Hosts make a lot of sense for small companies who can benefit from the aggregated demand and capital costs being spread over many clients.... but not when you're at the level of building your own datacenter, or even using a full rack.

It's funny how since 1980 people have been talking negatively about Apple as "vendor lock in". For most of that time it was advocating vendor lockin to windows.

The thing is when you build your system on an OS or hardware choice you're making "vendor lockin" to that platform. Build on Linux and you're locked into just Linux, unless you port.

There is little risk being "locked in" to the largest most successful company in the world. Plus the costs being dramatically lower given the rabidly higher performance of Apple's technology for this particular service more than covers the cost (in fact I think one Mac Pro probably replaces 4 or 5 Linux boxes doing this.)

If you think Optimized OpenGL shaders would do this, you're not understanding what it is that they are doing. You're just assuming it's a trivial problem, it is not.

Owning your hardware makes a great deal of sense when you are operating at scale.

Re: Racking Mac Pros

#248

Post by barbagallo »

skuhn wrote: post #190
frugalmail wrote: post #138

Earlier quoted context omitted.

> It really boils down to the fact that we want to use OS X, How the hell did you guys get funding to do this? I can't imagine any sane person wanting to put money behind this. Could I have their contact information?

The real question to me is: why would anyone fund doing this in EC2? Here's the quick math on cost per gflop, including all network and datacenter costs: Mac Pro: $5/gflop EC2 g2.xlarge: $21.19/gflop

It'll be REAL f'in expensive in EC2, that's for sure.

Re: Racking Mac Pros

#249

Post by jallmann »

MCRed wrote: post #247
jallmann wrote: post #78

This seems risky from a business perspective: it's voluntary vendor lock-in. What if Apple decides to change the Mac Pro form factor for the next iteration? Then you have to retool and are left with a bunch of incompatible chassis. What if Apple stagnates with hardware upgrades? You'd be stuck running obsolete hardware. What if Apple discontinues the entire Mac Pro line? Not to mention the price premium of Apple hard…

Why pay 5-10X as much to host on AWS? It's not for free. Hosts have a really nice markup, compared to hosting yourself. Hosts make a lot of sense for small companies who can benefit from the aggregated demand and capital costs being spread over many clients.... but not when you're at the level of building your own datacenter, or even using a full rack. It's funny how since 1980 people have been talking negatively abo…

Jeez, this post is brimming with strawmen. (Why am I even bothering...)

> Why pay 5-10X as much to host on AWS?

Nobody said anything about AWS...

> Hosts make a lot of sense for small companies

Sure, nobody is disputing their choice of colocating themselves.

> OS or hardware choice you're making "vendor lockin" to that platform

It is abundantly clear that the vendor lock-in refers to single sourcing your hardware. That problem is nonexistent on Windows, Linux, BSD, etc.

> I think one Mac Pro probably replaces 4 or 5 Linux boxes

Oh come on, now you're just talking crazy... see other posts in this thread for a cost/performance comparison.

> you're not understanding what it is that they are doing

On the contrary, I think I understand better than you. Do you perform a lot of image processing work on various platforms (including OSX and Linux)? I do.

Re: Racking Mac Pros

#250

Post by frugalmail »

ansible wrote: post #123
skuhn wrote: post #105

Earlier quoted context omitted.

The Tesla K80 didn't exist when I started this project, but to do some quick math: K80 gflop/s: 8740 2x FirePro D500 gflop/s: 3500 K80 runs about $4900 a card, whereas the entire Mac Pro (list price) is $4000. So it's 2.5x the performance at easily 2x the cost if not more. You're right that there is a cost advantage to going with commodity server hardware, but I don't think it's as great as most people think in this…

2x FirePro D500 gflop/s: 3500 That 3500 gflop/s for the D700? It is instead 2200 for the D500. http://www.amd.com/en-gb/solutions/workstations/d-series K80 runs about $4900 a card, whereas the entire Mac Pro (list price) is $4000. So it's 2.5x the performance at easily 2x the cost if not more. The 6GB VRAM version with the D700 costs another $600 USD each. The K80 has 12GB VRAM per GPU (24GB total per card). If your…

> it will be cheaper to rewrite the software for the render pipeline.

You don't even have to rewrite it, Linux imagemagick + OpenCV can handle the use cases of cropping and sizing trivially. They can keep the rest of the code (device mappings and CDN related I guess) unless that was implemented using ObjectiveC (this is another thing that I would think is crazy)

Post reply on HN