Live data from Hacker News

Pixar's Render Farm

twitter.com

161–170 of 323 posts

Re: Pixar's Render Farm

#161
post #71

In case someone is curious, the optimization they describe is trivial to do in Kubernetes - just enter a resource request for cpu without adding a limit.

kubernetes can't do it at this scale "trivially"

Firstly, K8s has no concept of licenses, it also is exceptionally weak on dependencies. A job graph for a VFX job can be well over 100k nodes, something that would crash k8s.

Secondly, tractor (https://rmanwiki.pixar.com/display/TRA/Tractor+2) is exceptionally fast at dispatching jobs to machines. I suspect its in the order of 50k a second, if not more.

Thirdly, getting k8s to talk to 25k machines without saturating the network is almost impossible.

fourthly, it doesn't do to well on "normal" network, try getting decent network throughput on one of K8s batshit networking schemes(each server on a farm will have at a minimum 2 10 gig links, more likley 2 40gig)

Re: Pixar's Render Farm

#162
post #59
post #47

Earlier quoted context omitted.

It's very painful to follow a conversation on Twitter. I'm not sure why they think the way they've done things makes sense.

It was never supposed to support conversation in the first place. People were supposed to shoot short, simple, single messages and other people maybe react to this with their own short, single messages.

"You are using our product wrong" is never the hallmark of a successful product manager.

Re: Pixar's Render Farm

#163

Earlier quoted context omitted.

maxing a CPU is easy, keeping it fed with data, and being able to save that data out is hard.

Yes, but the work units (frames) are large enough that I'm still surprised. Maybe they're not as parallelizable as I'd expect. E.g. if there's serial work to be done by reusing scene layout algorithms between frames.

A scene will have many thousands of assets (trees, cars, people, etc) each one will have the geo, which could be in the milllions of polygons (although they use sub-ds)

each "polygon" could have a 16k texture on it. You're pulling TBs of textures and other assets in each frame.

Re: Pixar's Render Farm

#164

From what I understand they still seem to render at 1080p and then upsample to 4k. Judging by Soul.

That seems extremely unlikely. The Renderman software they use has no issues rendering at 4k.

It's not really that unlikely. Most films render at 2k DCI. It's not so much that the software and hardware can't render higher, it's just diminishing gains for the increased render time.

Most 4k films till very recently actually have the digital elements at 2k DCI-ish resolutions and are upscaled. I can't speak to wether soul is rendered at 2k or 4k, but it wouldn't be surprising if it was 2k upscaled.

Re: Pixar's Render Farm

#165
post #3

My understanding (I am not an authority) is that for a long time, it has taken Pixar roughly an equal amount of time to render one frame of film. Something on the order of 24 hours. I don’t know what the real units are though (core-hours? machine-hours? simple wall clock?) I am not surprised that they “make the film fit the box”, because managing compute expenditures is such a big deal! (Edit: When I say "simple wall…

Maybe it’s society or maybe it’s intrinsic human nature, but there seems to be an overriding “only use resources to make it faster to a point, otherwise just make it better [more impressive?]”. Video games, desktop apps, web apps, etc. And now confirmed that it happens to movies at Pixar.

You can come at this from multiple directions.

On the one hand, it’s wise to only expend effort making something faster up to a point. At some point, unless a human has to wait for the result, there is no reason to make something faster [1].

On the other hand, once something takes more than a minute or two, and the person who started it goes and does something else, it doesn’t matter how long it takes, as long as it’s done before you get back. Film shots usually render overnight, so as long as they’re done in the morning and as long as they don’t prevent something else from being rendered by the morning, it doesn’t necessarily need to go faster. Somewhere out there is a blog post I remember about writing renderers and how artists behave; it posits perhaps there’s a couple of thresholds. If something takes longer than ten seconds to render, they’re going to leave to get coffee. If something takes longer than ten minutes to render, they’re going to start it at night and check on it in the morning.

[1] I always like the way Michael Abrash framed it:

“Understanding High Performance: Before we can create high-performance code, we must understand what high performance is. The objective (not always attained) in creating high- performance software is to make the software able to carry out its appointed tasks so rapidly that it responds instantaneously, as far as the user is concerned. In other words, high-performance code should ideally run so fast that any further improvement in the code would be pointless.

“Notice that the above definition most emphatically does not say anything about making the software as fast as possible. It also does not say anything about using assembly language, or an optimizing compiler, or, for that matter, a compiler at all. It also doesn’t say anything about how the code was designed and written. What it does say is that high-performance code shouldn’t get in the user’s way—and that’s all.” (From the “Graphics Programming Black Book”)

Re: Pixar's Render Farm

#166

From what I understand they still seem to render at 1080p and then upsample to 4k. Judging by Soul.

it's almost certainly rendered in cinema 4k (4,096 x 2,160 pixels), if not more. moreover it'll be in a 16/32bit log colourspace as well

Nobody renders higher than 4k DCI unless the film is screening on large format (imax etc).

The color space will be acesCG most likely these days.

Re: Pixar's Render Farm

#167
post #34

Oh man, I wanted this to contain much more details :( Whats the hardware? How much electric energy goes into rendering a frame or a whole movie? How do they provision it (as they keep #cores fixed)? They only talk about cores, do they even use GPUs? What's running on the machines? What did they optimize lately? So many questions! Maybe someone from Pixar's systems department is reading this :)?

> How do they provision it ex VFX sysadmin here. I'm not sure if they use their own scheduler or not. IF they do, they use tractor(might be tractor 2 now), which looks after putting the processes in the right places. Think K8s, but actually easy to use, well documented and reliable. (just not distributed, but then it scales way higher and is nowhere near as chatty) They would have a whole bunch of machines, some old…

Out of curiosity, did you move outside VFX and if yes to what industry, have you been enjoying and what motivated you?

Cheers

Re: Pixar's Render Farm

#169
post #34

Oh man, I wanted this to contain much more details :( Whats the hardware? How much electric energy goes into rendering a frame or a whole movie? How do they provision it (as they keep #cores fixed)? They only talk about cores, do they even use GPUs? What's running on the machines? What did they optimize lately? So many questions! Maybe someone from Pixar's systems department is reading this :)?

> How do they provision it ex VFX sysadmin here. I'm not sure if they use their own scheduler or not. IF they do, they use tractor(might be tractor 2 now), which looks after putting the processes in the right places. Think K8s, but actually easy to use, well documented and reliable. (just not distributed, but then it scales way higher and is nowhere near as chatty) They would have a whole bunch of machines, some old…

Rumour in the industry is that Pixar don't use Tractor themselves, and have a custom solution in Emeryville :)
Post reply on HN