Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Pixar's Render Farm

twitter.com

Post reply on HN
Print view View original

141–150 of 323 posts

Re: Pixar's Render Farm

#141

Post by dahart »

berkut wrote: post #125
dahart wrote: post #117

Earlier quoted context omitted.

> Because the expense is not really worth it I disagree with this takeaway. But full disclosure I’m biased: I work on OptiX. There is a reason Pixar and Arnold and Vray and most other major industry renderers are moving to the GPU, because the trends are clear and because it has recently become ‘worth it’. Many renderers are reporting factors of 2-10 for production scale scene rendering. (Here’s a good example: https…

I write a production renderer for a living :) So I'm well aware of the trade offs. As I mentioned, for lookdev and small scenes, GPUs do make sense currently (if you're willing the pay the penalty of getting code to work on both CPU and GPU, and GPU dev is not exactly trivial in terms of debugging / building compared to CPU dev). But until GPUs exist with > 64 GB RAM, for rendering large scale scenes, it's just not w…

I used to write a production renderer for a living, now I work with a lot of people who write production renderers for both CPU and GPU. I’m not sure what line you’re drawing exactly ... if you mean that it will take 3 or 4 years before the industry will be able to stop using CPUs for production rendering, then I totally agree with you. If you mean that it will take 3 or 4 years before industry can use GPUs for any production rendering, then that statement would be about 8 years too late. I’m pretty sure that’s not what you meant, so it’s somewhere in between there, meaning some scenes are doable on the GPU today and some aren’t. It’s worth it now in some cases, and not worth it in other cases.

The trend is pretty clear, though. The size of scenes than can be done on the GPU today is large and growing fast, both because of improving engineering and because of increasing GPU memory speed & size. It’s just a fact that a lot of commercial work is already done on the GPU, and that most serious commercial renderers already support GPU rendering.

It’s fair to point out that the largest production scenes are still difficult and will remain so for a while. There are decent examples out there of what’s being done in production with GPUs already:

https://www.chaosgroup.com/vray-gpu#showcase

https://www.redshift3d.com/gallery

https://www.arnoldrenderer.com/gallery/

Re: Pixar's Render Farm

#142

Post by chrisseaton »

dodobirdlord wrote: post #137
brundolf wrote: post #29

Earlier quoted context omitted.

In my (hobbyist) experience, path-tracing and rendering in general are enormously parallelizable. So if you can render X frames in parallel such that they all finish in 24 hours, that's roughly equivalent to saying you can render one of those frames in 24h/X. Of course I'm sure things like I/O and art-team-workflow hugely complicate the story at this scale, but I still doubt there's a meaningful concept of "wall-cloc…

Ray tracing is embarrassingly parallel, but it requires having most if not all of the scene in memory. If you have X,000 machines and X,000 frames to render in a day, it almost certainly makes sense to pin each render to a single machine to avoid having to do a ton of moving data around the network and in and out of memory on a bunch of machines. In which case the actual wall-clock time to render a frame on a single…

Exactly - move the compute to the data, not the data to the compute.

Re: Pixar's Render Farm

#143

Post by jack2222 »

tinco wrote: post #138
aprdm wrote: post #75

Earlier quoted context omitted.

Not Pixar specifically but Modern VFX and Animation studios usually have a bare metal render farm, they usually are pretty beefy -- think at least 24 cores / 128 GB of RAM per node. Usually in crunch time if there's not enough nodes in the render farm they might rent nodes connecting them to their network for a period of time, or they might use the cloud, or they might get budget to increase their render farms. From…

Can confirm cloud GPU is way overpriced if you're doing 24/7 rendering. We run a bare metal cluster (not VFX but photogrammetry) and I pitched our board on the possibilities. I really did not want to run a bare metal cluster, but it just does not make sense for a low margin startup to use cloud processing. Running 24/7 for three months, it's cheaper to buy consumer grade hardware with similar (probably better) perfor…

I've had renders cost $50,000! Or CTO was less than amused

Re: Pixar's Render Farm

#144

Post by dodobirdlord »

dagmx wrote: post #62
supernova87a wrote: post #53

I would love to know about some curious questions, for example: If there's a generally static scene with just characters walking through it, does the render take advantage of rendering the static parts for the whole scene once , and then overlay and recompute the small differences caused by the moving things in each individual sub frame? Or, alternatively what "class" of optimizations does something like that fall in…

If you're interested in production rendering for films, there's a great deep dive into all the major studio renderers https://dl.acm.org/toc/tog/2018/37/3 As for your questions: > Is rendering of video games more similar to rendering for movies, or VFX? This question is possibly based on an incorrect assumption that feature (animated) films are rendered differently than VFX. They're identical in terms of most tech st…

> This question is possibly based on an incorrect assumption that feature (animated) films are rendered differently than VFX. They're identical in terms of most tech stacks including rendering and the process is largely similar overall.

Showcased by the yearly highlights reel that the Renderman team puts out.

https://vimeo.com/388365999

Re: Pixar's Render Farm

#145

Post by capableweb »

klodolph wrote: post #60
ChuckNorris89 wrote: post #9

Earlier quoted context omitted.

Wait, what? 24 hours per frame?! At the standard 24fps it takes you 24 days per film second which works out to 473 years for the average 2 hour long film which can't be right.

Again, I'm not sure whether this is core-hours, machine-hours, or wall clock. And to be clear, when I say "wall clock", what I'm talking about is latency between when someone clicks "render" and when they see the final result. My experience running massive pipelines is that there's a limited amount of parallelization you can do. It's not like you can just slice the frame into rectangles and farm them out.

> It's not like you can just slice the frame into rectangles and farm them out.

Funny thing, you sure can! Distributed rendering of single frames been a thing for a long time already.

Re: Pixar's Render Farm

#146

Post by lhoff »

berkut wrote: post #139
foota wrote: post #135

Earlier quoted context omitted.

Given current consumer GPUs are at 24 GB I think 3-4 years is likely overly pessimistic.

They've been at 24 GB for two years though - and they cost an arm and a leg compared to a CPU with a similar amount. It's not just about them existing, they need to be cost effective.

Not anymore. The new Ampere based Quadros and Teslas just launched with up to 48GB of RAM. A special datacenter version with 80Gb is also already announced: https://www.nvidia.com/en-us/data-center/a100/

They are really expensive though. But chassis and rackspace also isn't free. If one beefy node with a couple GPUs can replace have a rack of CPU only Nodes the GPUs are totally worth it.

I'm not too familiar with 3D rendering but in other workloads the GPU speedup is so huge that if its possible to offload to the GPU it made sense to do it from a economical perspective.

Re: Pixar's Render Farm

#148

Post by Finnucane »

iambateman wrote: post #94
lmilcin wrote: post #59

Earlier quoted context omitted.

It was never supposed to support conversation in the first place. People were supposed to shoot short, simple, single messages and other people maybe react to this with their own short, single messages.

We are way way past that point. It’s time for Twitter to evolve in so many ways.

Evolve or die. Twitter dies is a good option.

Re: Pixar's Render Farm

#149

Post by 2OEH8eoCRo0 »

dodobirdlord wrote: post #147
2OEH8eoCRo0 wrote: post #93

From what I understand they still seem to render at 1080p and then upsample to 4k. Judging by Soul.

That seems extremely unlikely. The Renderman software they use has no issues rendering at 4k.

The 4k streaming copy of the film has stairsteps though. Like it's been upsampled. I'm sure their software can render at 4k but they choose not to for whatever reason.

Re: Pixar's Render Farm

#150

Post by KaiserPro »

nom wrote: post #34

Oh man, I wanted this to contain much more details :( Whats the hardware? How much electric energy goes into rendering a frame or a whole movie? How do they provision it (as they keep #cores fixed)? They only talk about cores, do they even use GPUs? What's running on the machines? What did they optimize lately? So many questions! Maybe someone from Pixar's systems department is reading this :)?

> How do they provision it

ex VFX sysadmin here. I'm not sure if they use their own scheduler or not. IF they do, they use tractor(might be tractor 2 now), which looks after putting the processes in the right places. Think K8s, but actually easy to use, well documented and reliable. (just not distributed, but then it scales way higher and is nowhere near as chatty)

They would have a whole bunch of machines, some old some new, some with extra memory, for particle sims, some with extra cores for just plain rendering. Each machine will be separated into slots, which are made up of a fixed number of cores. Normally memory is guarded but CPU is not (ie, you only get 8 gigs of ram, but as much CPU as you can consume. Context switching the CPU is fast, memory not so much.) I'm not sure on how pixar does it, but at a large facility like ilm/framestore/dneg the farm will be split into shows, with guaranteed minimum allocation of cores. this is controlled by the scheduler. crucially it'll be over subscribed, so jobs are ordered by priority.

As for actual hardware provisioning, thats quite cool. In my experience there will be a bringup script that talks to the iLo/iDrac/other management system. When a machine is plugged in, it'll be seen by the bringup script, download the xml/config/other goop that tells the bios how to configure and boot from the network, connect to the imaging system and install whatever version of linux they have.

As for power per frame, each frame will be made up of different plates, so if you have a water sim, that'll be rendered separately, along with other assets. These can then be combined afterwards in nuke to tweak and make pretty without having to render everything again.

That being said, a crowd shot with lots of characters with hair, or a water/smoke/ice effect can take 25+ hours per frame to render. So think a 100core/thread machine redlining for 25 hours, plus a few hundreds TB of spinny disk. (then it'll be tweak 20ish times)

optimisations wise, I suspect it's mostly getting software to play nice on the same machine, or beating TDs to make better use of assets, or adjusting the storage to make sure its not being pillaged too much.

Post reply on HN