I worked on this movie, I was at DNEG at the time. One of the standout things that I remember is that this particular simulation was toxic to the fileserver that it was being stored on.
From what I recall, I don't think that it was running on that many machines at once. Mainly because it required the high memory nodes that were expensive. I think it was only running on ~10 possibly 50 machines concurrently. But I could be wrong.
What it did have was at least one dedicated fileserver though. Each of the file servers at the time were some dual proc dell 1u thing with as much ram as you could stuff in them at the time (384 gigs I think). They were attached by SAS to a single 60 drive 4u raid array. (Dell PowerVault MD3460 or something along those lines. They are rebadged by Dell and were the first practical hotswap enclosure that took normal 3.5" SAS drives, that didn't cost the earth)
The array was formatted into 4 raid6 groups, and LVM'd together on the server. it was then shared out by NFS over bonded 10gig links.
Anyway. That simulation totally fucked the disks in the array. By the time it finished (I think it was a 2 week run time) it had eaten something like 14 hard drives. Every time a new disk was inserted, another would start to fail. It was so close to fucking up the whole time.
I had thought that the simulation was a plugin for houdini, or one of the other fluid simulation engines we had kicking around, rather than a custom 40k C++ program.