Racking Mac Pros • Hacker News Board

Re: Racking Mac Pros

#271

Post by x0054 » Thu, May 07, 2015, 3:56 AM UTC

barkingcat wrote: post #262

x0054 wrote: post #242

Earlier quoted context omitted.

It strikes me that OS X is a solid server platform. Linux is always a more flexible choice, but if OS X works for you, it works for you. That being said, is sticking cylindrical mac pros on their side into square racks really the best solution. There are 2 problems I see with this design: 1: You are placing the Mac Pros on their side, which may lead to premature bearing failure on the main cooling fan. Apple designed…

"combined with the fact that it would be simply stupid for a company who makes their own OS to run anything but that OS" Why would you assume that? There are a ton of things that linux does better than OS X - and it would be extremely stupid for any company regardless of size to not use the right tool for the job. For example, even IBM uses, sells, and supports Linux instead of AIX or OS/360 on their line of servers…

At it's core OS X is Unix. In what way would Linux be a better choice? I am not saying that Linux is a worse choice, but for a company that writes an OS as one of it's core businesses, it only makes sense to run that OS in as many places as possible. For one, by running OS X as a server OS they would necessarily spend more time on development and improvement of the OS core. This would pay off in the long run by further improving stability and reliability of OS X.

I am not arguing that OS X is the perfect solution in most circumstances, but it can be a good solution in many situations, especially if you are Apple, and have the full source and the capability to adopt the OS as necessary.

Microsoft, especially nowadays, tries to be very cross compatible, so it's not surprising that Azure supports Linux apps and guests. But Azure RUNS on Windows Server 2008, not Linux, not Unix.

Re: Racking Mac Pros

#272

Post by frugalmail » Thu, May 07, 2015, 4:09 AM UTC

skuhn wrote: post #190

frugalmail wrote: post #138

Earlier quoted context omitted.

> It really boils down to the fact that we want to use OS X, How the hell did you guys get funding to do this? I can't imagine any sane person wanting to put money behind this. Could I have their contact information?

The real question to me is: why would anyone fund doing this in EC2? Here's the quick math on cost per gflop, including all network and datacenter costs: Mac Pro: $5/gflop EC2 g2.xlarge: $21.19/gflop

Not sure where you got ec2 out of my comment.

I also think you need to redo your math on the price per gflop for a Mac pro, ypou seem to be at least half the price of my back of the envelope work. Unless you have some crazy good supplier.

Re: Racking Mac Pros

#273

Post by ameen » Thu, May 07, 2015, 4:16 AM UTC

skuhn wrote: post #156

jallmann wrote: post #112

Earlier quoted context omitted.

> look at it as an entry barrier that protects them against competitors too What barrier to entry? Their customers don't care that OSX is running under the hood. You can offer an image processing service using any platform today. Sure, on Linux it probably wouldn't be as efficient, but it doesn't have to be. Scaling is a Good Problem to have. Basically, as you grow, it helps to take a critical look at risk factors an…

(I'm the datacenter manager at imgix, and I wrote the article) I completely agree with your concerns, and I'm constantly evaluating our business for operational risks and inefficiencies. There's a lot of stuff that I can't share in public about this, but what I can say is that the math works out (for now): OS X graphics processing is worth the downsides. It may not always be the case, and we're built in a flexible wa…

> OS X graphics processing is worth the downsides

How different is it? Aren't they dependent on OpenGL and Nvidia/AMD GPUs/Drivers themselves? Wouldn't it make better sense/efficient to invest in becoming platform agnostic and optimize this.

I only say so cause it seems like Imgix could massively benefit from such a move and maybe look into other solutions which you currently can't consider (Custom ARM silicon - PowerVR-based servers, Professional AMD/Nvidia GPUs, etc)

Re: Racking Mac Pros

#274

Post by allochthon » Thu, May 07, 2015, 4:19 AM UTC

I didn't know about the cylindrical Mac Pros. They are quite attractive.

Re: Racking Mac Pros

#275

Post by headShrinker » Thu, May 07, 2015, 4:23 AM UTC

grecy wrote: post #192

rsync wrote: post #109

I have two conflicting responses to what I am seeing here ... First, this is awesome. Just like I want to live in a world where people are paying picodollars for cloud storage[1], I also want to live in a world where a bunch of mac pro cylinders are racked up in a datacenter. Very cool. Second, this is complete silliness. I'm not going to go down the rabbithole of flops per dollar, but there is no way that you can't…

> there is no way that you can't build a hackintosh Have you ever personally run a Hackintosh, full-time for a prolonged period of time? It's anecdotal, but I can assure that once you're used to how OS X and the Apple hardware work together and never, ever, ever crash, using a Hackintosh is an exercise in frustration. I had one of the known-best Hackintosh configurations in existence, and it didn't hold a candle to t…

I'm surprised at how much attention 'hackintosh' is getting in this thread. It's a completely naive sub-topic. If you are a US corporation 'hachintosh' is completely taboo, beyond taboo, it's illegal. If hachintosh is how some cooperations run, yikes, let me know so I can never be their customer. If a company is that cheap with their hardware and their morals, I would hate to see how they treat their employees. (It's also naive to think if a company saves money on their hardware by hacking the shit out of it, that money saved will be siphoned in to workers paychecks.)

Re: Racking Mac Pros

#276

Post by kibibu » Thu, May 07, 2015, 4:30 AM UTC

rbanffy wrote: post #256

MCRed wrote: post #245

Earlier quoted context omitted.

No, they wouldn't. Look at the current state of Mackintoshes. People are having kernel panics and struggling to keep their machines running with current software. OS X moves pretty fast, possibly faster than Linux, and Apple builds it to support Mac hardware.... the teams who are porting hackintosh code have to support a lot more hardware variety and they have less resources than, say, linux. Running mackintoshes in…

> People are having kernel panics and struggling to keep their machines running with current software. Not here. My Mac is at about 11 days of uptime and it's under constant use. At this moment, I can't say it's less reliable then my Linux machines. In this specific case, however, I'd consider ditching the enclosure and ducting cold air through the internal chassis/heatsink. A Macpro is, essentially a heatsink with b…

I think the part you quoted was referring to hackintosh machines only.

Still, there's no doubt in my mind that Apple are doing some "move fast and break things" OS development.

Re: Racking Mac Pros

#277

Post by skuhn » Thu, May 07, 2015, 4:34 AM UTC

joshmn wrote: post #269

skuhn wrote: post #234

Earlier quoted context omitted.

The Minis are incredibly simple machines inside, and their airflow is not great. You could pop their logic boards out and run them without the chassis. With some thought it wouldn't be too tough to significantly improve their airflow for this environment. I do have an existing rack design that holds 64 of them (and other people have gone denser, with operational compromises I prefer not to make), so there's no great…

Thanks for the detailed response. Really cool stuff you guys have going on. What other companies utilize Apple hardware in this way at this kind of scale? While not "out of this world" in comparison to some of the big players who have tackled scaling, it's definitely significant considering Apple hardware.

I'm not aware of anyone who specifically uses Mac Pros (although I've heard some private rumblings). I suspect part of the issue is that the old form factor was not very rack-friendly, and people haven't gotten comfortable with the new form factor. Maybe this chassis design will help move this forward.

Mac Minis are a little more common than it might seem at first glance, particularly for use cases that some other people have outlined in their comments throughout this thread. Mozilla uses them to test Firefox builds on OS X for instance. I would imagine that places like Sauce Labs must have a Mac Mini farm to facilitate browser tests on OS X.

I'm not aware of any other service that operates in the same space as imgix that runs outside of EC2, so they definitely aren't using OS X there. I think in general there's a sort of disregard for the particular graphics processing benefits that OS X provides (as evidenced by some of the comments in the thread).

I would also be remiss to not mention Mac Mini Colo (http://macminicolo.net/) who do co-located hosting. imgix started out with them, and they did a great job.

There's another interesting use case where you need to have OS X (or iOS): when you want to display photos taken on iOS devices with their applied filters (the images are stored pristine, and the filters are applied on top when you view them). To recreate these photos exactly as they were on the device, you ideally need to render it within an Apple environment. You can probably imagine the use case for a service that stores a lot of user generated photography, in a world where iPhones are the most popular cameras (https://www.flickr.com/cameras).

I also heard through the grapevine today that a certain film studio is interested in getting one of these chassis to test out, because they saw this article. That's pretty exciting to hear, even though we don't profit in any way from the sale of these chassis.

Re: Racking Mac Pros

#278

Post by skuhn » Thu, May 07, 2015, 4:39 AM UTC

ameen wrote: post #273

skuhn wrote: post #156

Earlier quoted context omitted.

(I'm the datacenter manager at imgix, and I wrote the article) I completely agree with your concerns, and I'm constantly evaluating our business for operational risks and inefficiencies. There's a lot of stuff that I can't share in public about this, but what I can say is that the math works out (for now): OS X graphics processing is worth the downsides. It may not always be the case, and we're built in a flexible wa…

> OS X graphics processing is worth the downsides How different is it? Aren't they dependent on OpenGL and Nvidia/AMD GPUs/Drivers themselves? Wouldn't it make better sense/efficient to invest in becoming platform agnostic and optimize this. I only say so cause it seems like Imgix could massively benefit from such a move and maybe look into other solutions which you currently can't consider (Custom ARM silicon - Powe…

Without going into too much detail about our stack, there's quite a bit more to Core Image than OpenGL+graphics card driver.

Those are important components, and we're not talking about splitting the atom here, but Apple has had a number of smart people working on graphics technologies for a long time now. imgix also has a bunch of smart people working on this, but for a much shorter period of time.

Re: Racking Mac Pros

#279

Post by gh02t » Thu, May 07, 2015, 4:42 AM UTC

x0054 wrote: post #271

barkingcat wrote: post #262

Earlier quoted context omitted.

"combined with the fact that it would be simply stupid for a company who makes their own OS to run anything but that OS" Why would you assume that? There are a ton of things that linux does better than OS X - and it would be extremely stupid for any company regardless of size to not use the right tool for the job. For example, even IBM uses, sells, and supports Linux instead of AIX or OS/360 on their line of servers…

At it's core OS X is Unix. In what way would Linux be a better choice? I am not saying that Linux is a worse choice, but for a company that writes an OS as one of it's core businesses, it only makes sense to run that OS in as many places as possible. For one, by running OS X as a server OS they would necessarily spend more time on development and improvement of the OS core. This would pay off in the long run by furth…

Because it isn't really about the OS, it's about the software. OS X is fine as a server platform, but it doesn't have the same software and support ecosystem for data center usage. Apple dumped that market with the Xserve because it didn't work for them.

Red Hat/Suse/Oracle etc. all sell tailored solutions for that usage that are Linux specific technology (mostly, some stuff gets ported to other Unix derivatives but most doesn't). Sure Apple could do all that too, but they don't want to. It isn't their market, so why sink money and effort into engineering OS X to do it when they can just buy high quality products ready to go?

Re: Racking Mac Pros

#280

Post by skuhn » Thu, May 07, 2015, 4:43 AM UTC

frugalmail wrote: post #272

skuhn wrote: post #190

Earlier quoted context omitted.

The real question to me is: why would anyone fund doing this in EC2? Here's the quick math on cost per gflop, including all network and datacenter costs: Mac Pro: $5/gflop EC2 g2.xlarge: $21.19/gflop

Not sure where you got ec2 out of my comment. I also think you need to redo your math on the price per gflop for a Mac pro, ypou seem to be at least half the price of my back of the envelope work. Unless you have some crazy good supplier.

Exposing more detail behind this math is unfortunately not something that I'm ready to do, but I'm pretty comfortable with it in broad strokes. EC2 really is that much more expensive, when you factor in things like network bandwidth.

As I noted elsewhere, I mention EC2 because all of our (funded) competitors run there. We can split hairs over whether I could save 10% on Linux systems vs Mac systems, but the elephant in the room are all of the companies trying to make this sort of service work in EC2. You can't do it, and make money at the same time. Even if you can make money at small scale, you will eventually be crushed by your own success.

My overriding goal for imgix's datacenter strategy (and elsewhere in the company) is to build for success. To do that, we have to get the economies of scale right. I believe we have done so.