This really makes me want to work for Pixar.
Pixar's Render Farm
261–270 of 323 posts
Re: Pixar's Render Farm
#262Can anyone comment on why Pixar uses standard CPU for processing instead of custom hardware or GPU? I'm wondering why they haven't invested in FPGA or completely custom silicon that speeds up common operations by an order of magnitude. Is each show that different that no common operations are targets for hardware optimization?
Because the expense is not really worth it - even GPU rendering (while around 3/4 x faster than CPU rendering) is memory constrained compared to CPU rendering, and as soon as you try and go out-of-core on the GPU, you're back at CPU speeds, so there's usually no point doing GPU rendering for entire scenes (which can take > 48 GB of RAM for all geometry, accel structures, textures, etc) given the often large memory re…
> while around 3/4 x faster than CPU rendering
My understanding is that for neural networks, the speedup is much more than 4x. Does anyone know why there's such a difference?
Re: Pixar's Render Farm
#263Earlier quoted context omitted.
Can confirm cloud GPU is way overpriced if you're doing 24/7 rendering. We run a bare metal cluster (not VFX but photogrammetry) and I pitched our board on the possibilities. I really did not want to run a bare metal cluster, but it just does not make sense for a low margin startup to use cloud processing. Running 24/7 for three months, it's cheaper to buy consumer grade hardware with similar (probably better) perfor…
How do you manage the bare metal cluster? (E.g. apt/yum updates but also networking and such)
I'm a bit out of date but if we are talking about rendering (not data retrieval workloads) I believe the best way is fundamentally the same as it was 25 years ago: network boot, mostly network storage, and applying local config overlays based on MAC address or equivalents. Exactly what push or pull techniques are in vogue I am not sure but definitely no running package managers on each node. You want as little as possible locally -- just a scratchpad disk that can be rebuilt automatically in minutes.
Re: Pixar's Render Farm
#264Earlier quoted context omitted.
it's almost certainly rendered in cinema 4k (4,096 x 2,160 pixels), if not more. moreover it'll be in a 16/32bit log colourspace as well
Have you examined it or are you guessing? Because I'd have assumed they did too but I see jagged stairsteps on the 4k stream when zooming on sharp edges.
There is no way on earth that they will render at 1080. If it was 2007 it might have been 2k.
The master will have been rendered into a DCP.
What gets delivered to the streaming service is another thing.
what then ends up at your screen is even more of a guess.
A 4K dcp is a JPEG2000 stream[1], with 16bit colour. Something like 60-120 megabytes a second. Obviously this isn't practical to stream to consumers. 4K over a stream is always a balancing act, more often than not it drops to 1080.
[1] well normally it is
Re: Pixar's Render Farm
#265Earlier quoted context omitted.
I spent tenish years in VFX. I moved away in 2014, because the hours and pay were abysmal. I still love the industry. I moved to a large profitable financial news paper, which had cute scaling issues (ie they were all solved, so engineers tried to find new and interesting ways to unsolved them ) I then moved to a startup that made self building machine readable maps, which allowed me to play with scale again, but on…
Interesting, thanks for sharing! I've been in VFX around ~6 years and was in the sw industry before (and the hw industry before it). I find VFX really fun as far as job! Sometimes I do think about leaving mostly for Pay reasons but the pay has been decent enough recently (basically FAANG base pay without RSU/Bonus...). It is interesting how we have a lot of big scale problems that goes unrecognized, I find the proble…
MAybe we will work together soon!
Re: Pixar's Render Farm
#266Earlier quoted context omitted.
I spent tenish years in VFX. I moved away in 2014, because the hours and pay were abysmal. I still love the industry. I moved to a large profitable financial news paper, which had cute scaling issues (ie they were all solved, so engineers tried to find new and interesting ways to unsolved them ) I then moved to a startup that made self building machine readable maps, which allowed me to play with scale again, but on…
Were you at The Foundry (I think I know who you are)? If so, I think we were both there at the same time!
If you worked on katana then I think I know who you are too!
Re: Pixar's Render Farm
#267Earlier quoted context omitted.
Given current consumer GPUs are at 24 GB I think 3-4 years is likely overly pessimistic.
They've been at 24 GB for two years though - and they cost an arm and a leg compared to a CPU with a similar amount. It's not just about them existing, they need to be cost effective.
Desktop GPUs could have 64GB of GDDR right now but the memory bus width to drive those bits optimally (in primary use case of real-time game rendering, not offline) would up the power and heat dissipation requirements beyond what is currently engineered onto a [desktop] PCIE card.
If 8k gaming becomes a real thing you can expect work to be done towards a solution, but until then not so much.
Edit: added [desktop] preceding PCIE
Re: Pixar's Render Farm
#268Earlier quoted context omitted.
They've been at 24 GB for two years though - and they cost an arm and a leg compared to a CPU with a similar amount. It's not just about them existing, they need to be cost effective.
Not anymore. The new Ampere based Quadros and Teslas just launched with up to 48GB of RAM. A special datacenter version with 80Gb is also already announced: https://www.nvidia.com/en-us/data-center/a100/ They are really expensive though. But chassis and rackspace also isn't free. If one beefy node with a couple GPUs can replace have a rack of CPU only Nodes the GPUs are totally worth it. I'm not too familiar with 3D…
Hashing and linear algebra kernels get much more speedup on a GPU than a vfx pipeline does. But I am glad to see reports here detailing that the optimization of vfx is progressing.
Re: Pixar's Render Farm
#269Earlier quoted context omitted.
What do you mean by a lot more threads? Are you comparing an epyc?
Yeah, that came off clumsily (I’d lost part of my comment while switching tabs on my phone). An AMD Rome/Milan part will give you 256 decent threads on a 2S box with a ton of RAM for say $20-25k at list price (e.g., a Dell power edge without any of their premium support or lots of flash). By comparison, the list price of just an A100 is $15k (and you still need a server to drive the thing). So for shops shoving these…
Is the high priced 256 thread part that interesting for rendering? You can get 4 of the 64 thread parts on separate boards and each one will have its own 8 channel ddr instead of having to share that bandwidth. Total performance will be higher for less or same money. Power budget will be higher but only a couple dollars a day, at most. But I haven't been involved in a cluster for some time, so not really sure what is done these days.
Re: Pixar's Render Farm
#270Earlier quoted context omitted.
> It's not like you can just slice the frame into rectangles and farm them out. Funny thing, you sure can! Distributed rendering of single frames been a thing for a long time already.
What about GI? You can't just slice GI into pieces.
This has been possible even for CGI tinkerers like me with C4D for more than ten years.