Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Inside the Titan Supercomputer: 299K AMD x86 Cores and 18.6K Nvidia GPU Cores

anandtech.com

Post reply on HN
Print view View original

11–20 of 29 posts

Re: Inside the Titan Supercomputer: 299K AMD x86 Cores and 18.6K Nvidia GPU Cores

#11

Post by caf »

paulsutter wrote: post #7

Does anyone know why they have a separate disk IO system when they could more easily just plug drives into each node/motherboard for higher aggregate throughout, less complexity, and a lower overall cost? EDIT: Blade systems or no, the drives have to physically be placed somewhere. Having a separate subsystem can only take up more space, not less. Two reasons I can think of: (1) independent scaling of compute and sto…

If you look through the gallery you'll see that their disk subsystem is using a distributed filesystem, Lustre (http://www.lustre.org/).

Re: Inside the Titan Supercomputer: 299K AMD x86 Cores and 18.6K Nvidia GPU Cores

#12

Post by paulsutter »

caf wrote: post #11
paulsutter wrote: post #7

Does anyone know why they have a separate disk IO system when they could more easily just plug drives into each node/motherboard for higher aggregate throughout, less complexity, and a lower overall cost? EDIT: Blade systems or no, the drives have to physically be placed somewhere. Having a separate subsystem can only take up more space, not less. Two reasons I can think of: (1) independent scaling of compute and sto…

If you look through the gallery you'll see that their disk subsystem is using a distributed filesystem, Lustre ( http://www.lustre.org/ ).

They would need something much more like GFS than Lustre. Lustre is designed for the exact approach they're using.

Re: Inside the Titan Supercomputer: 299K AMD x86 Cores and 18.6K Nvidia GPU Cores

#13

Post by willvarfar »

paulsutter wrote: post #7

Does anyone know why they have a separate disk IO system when they could more easily just plug drives into each node/motherboard for higher aggregate throughout, less complexity, and a lower overall cost? EDIT: Blade systems or no, the drives have to physically be placed somewhere. Having a separate subsystem can only take up more space, not less. Two reasons I can think of: (1) independent scaling of compute and sto…

> Does anyone know why they have a separate disk IO system

It would be very interesting if they could describe all their general design decisions, such as this

> ... when they could more easily just plug drives into each node/motherboard for higher aggregate throughout, less complexity, and a lower overall cost?

doesn't the fact that they haven't put the drives on the compute nodes make you question your claim that it would have been 'easier' and 'better' and 'cheaper'?

Re: Inside the Titan Supercomputer: 299K AMD x86 Cores and 18.6K Nvidia GPU Cores

#14

Post by batgaijin »

tmurray wrote: post #6
hendzen wrote: post #2

This is pretty awe-inspiring but as a programmer I know it would be fairly difficult to use this machine for existing workloads because so much code would have to be rewritten from typical x86 code to CUDA/OpenCL to use all those GPUs. Personally, I'm more excited for the next wave of supercomputers built with racks of Xeon Phis [1]. [1] - http://www.intel.com/content/www/us/en/high-performance-comp...

(full disclosure: used to work for NV on CUDA and did very extensive work on Titan, so I am probably biased) If you think your existing MPI app is going to automatically scale to a heterogeneous architecture (high-power x86 on the main CPU, Xeon Phi cores on the accelerator) and get acceptable performance, sorry, it's not going to happen. The fundamental constraints on 2012/2013 Xeon Phi performance that determine ho…

To my understanding GPU's basically suck at anything with decision paths/move away from straight matrix manipulation/signals analysis, right?

Re: Inside the Titan Supercomputer: 299K AMD x86 Cores and 18.6K Nvidia GPU Cores

#15

Post by dfox »

paulsutter wrote: post #7

Does anyone know why they have a separate disk IO system when they could more easily just plug drives into each node/motherboard for higher aggregate throughout, less complexity, and a lower overall cost? EDIT: Blade systems or no, the drives have to physically be placed somewhere. Having a separate subsystem can only take up more space, not less. Two reasons I can think of: (1) independent scaling of compute and sto…

Common programming model of parallel supercomputers expect something like one shared filesystem across all nodes (used for input and output and checkpointing and such, almost never as temporary per-node storage). And most readily available and scalable enough implementations of that expect exactly this architecture as it is easier to implement in software, reason about it's performance and also significantly cheaper and easier to maintain.

Re: Inside the Titan Supercomputer: 299K AMD x86 Cores and 18.6K Nvidia GPU Cores

#16

Post by [deleted] »

batgaijin wrote: post #14
tmurray wrote: post #6

Earlier quoted context omitted.

(full disclosure: used to work for NV on CUDA and did very extensive work on Titan, so I am probably biased) If you think your existing MPI app is going to automatically scale to a heterogeneous architecture (high-power x86 on the main CPU, Xeon Phi cores on the accelerator) and get acceptable performance, sorry, it's not going to happen. The fundamental constraints on 2012/2013 Xeon Phi performance that determine ho…

To my understanding GPU's basically suck at anything with decision paths/move away from straight matrix manipulation/signals analysis, right?

[deleted]

Re: Inside the Titan Supercomputer: 299K AMD x86 Cores and 18.6K Nvidia GPU Cores

#17

Post by [deleted] »

batgaijin wrote: post #14
tmurray wrote: post #6

Earlier quoted context omitted.

(full disclosure: used to work for NV on CUDA and did very extensive work on Titan, so I am probably biased) If you think your existing MPI app is going to automatically scale to a heterogeneous architecture (high-power x86 on the main CPU, Xeon Phi cores on the accelerator) and get acceptable performance, sorry, it's not going to happen. The fundamental constraints on 2012/2013 Xeon Phi performance that determine ho…

To my understanding GPU's basically suck at anything with decision paths/move away from straight matrix manipulation/signals analysis, right?

[deleted]

Re: Inside the Titan Supercomputer: 299K AMD x86 Cores and 18.6K Nvidia GPU Cores

#20

Post by confluence »

batgaijin wrote: post #14
tmurray wrote: post #6

Earlier quoted context omitted.

(full disclosure: used to work for NV on CUDA and did very extensive work on Titan, so I am probably biased) If you think your existing MPI app is going to automatically scale to a heterogeneous architecture (high-power x86 on the main CPU, Xeon Phi cores on the accelerator) and get acceptable performance, sorry, it's not going to happen. The fundamental constraints on 2012/2013 Xeon Phi performance that determine ho…

To my understanding GPU's basically suck at anything with decision paths/move away from straight matrix manipulation/signals analysis, right?

GPUs suck at any problem that cannot be easily divided. If you can map a function over arbitrary chunks of self-contained data GPUs will perform better.
Post reply on HN