Live data from Hacker News

Pixar's Render Farm

twitter.com

271–280 of 323 posts

Re: Pixar's Render Farm

#271
post #151

Earlier quoted context omitted.

The 4k streaming copy of the film has stairsteps though. Like it's been upsampled. I'm sure their software can render at 4k but they choose not to for whatever reason.

are you sure that's not just an artifact of it being streamed?

probably. Looking at streaming 4k is just sad, as this is just wasting money (on devices and bandwidth) for marketing having abyssmal quality at 10-20Mbit/s. With a FHD-bluray weighing in at 30-50MBit/s that's nothing to wonder about though I guess..

Re: Pixar's Render Farm

#272
post #262
post #54

Earlier quoted context omitted.

Because the expense is not really worth it - even GPU rendering (while around 3/4 x faster than CPU rendering) is memory constrained compared to CPU rendering, and as soon as you try and go out-of-core on the GPU, you're back at CPU speeds, so there's usually no point doing GPU rendering for entire scenes (which can take > 48 GB of RAM for all geometry, accel structures, textures, etc) given the often large memory re…

> while around 3/4 x faster than CPU rendering My understanding is that for neural networks, the speedup is much more than 4x. Does anyone know why there's such a difference?

Sure. Training neural nets is somewhat analogous to starting on the top of a mountain looking for the lowest of the low points of the valley below. But instead of being in normal 3d space you might have 1000d determining your altitude, so you can't see where you're going, and you have to iterate and check. But ultimately you just calculate the same chain of the same type of functions over and over until you've reached a pretty low point in the hypothetical valley.

OTOH, Vfx rendering involves a varying scene with moving light sources, cameras, objects, textures, and physics. Much more dynamic interactions. This is a gross simplification but I hope it helps.

Re: Pixar's Render Farm

#273
I was a render wrangler and sys admin at a vfx shop for a few years, I then moved to a digital intermediate shop. Reading this thread has me remembering fun times.

I wrote about my experiences in more detail a couple of months ago:

https://blog.markjgsmith.com/2020/11/24/what-its-like-workin...

There’s also some render farm setup stuff in my portfolio on the blog if you’re interested.

The hardware side of things was incredible. So much expensive kit. There’s a lot of great hardware related comments in the thread, so I won’t go over that.

The thing that I found really cool was how all the software systems were setup for so many artists to collaborate. Though we didn’t talk about agile methodologies (it was back in 2003/2004), the ways we worked had a lot of similarity to a classic agile / scrum setup, with daily standups, though they were called ‘dailies’, and there was no notion of sprints (we were always sprinting!) or sprint planning, but similar planning was done by producers and department heads, instead of user stories / features, people worked on ‘shots’. At the digital intermediate place the unit of work tended to be reels since we were scanning and printing entire reels, combining all the shots that the vfx houses completed.

Everyone used version control, though it was subversion, I don’t think git was that popular back then. Artists worked on their shots, checking in their project files rather than rendered files, we could always re-render a shot if necessary. There were also cli tools to submit finished rendered files, which were automatically organised into a standard folder structure on shared storage. The artists Linux shells would load environment variables from a dB, and their applications (Shake/Maya/Houdini etc) would load these transparently, so they never had to worry about where things were stored. That was all automatic as long as they knew which shot they were working on.

It was a great place to learn about technology and collaborating on digital production at scale.

I’ve always thought there were a lot of setups and tools that could be applied to software development at scale. I’d love to work on that sort of project. Hint: I am available for hire :)

Re: Pixar's Render Farm

#274
post #138
post #75

Earlier quoted context omitted.

Not Pixar specifically but Modern VFX and Animation studios usually have a bare metal render farm, they usually are pretty beefy -- think at least 24 cores / 128 GB of RAM per node. Usually in crunch time if there's not enough nodes in the render farm they might rent nodes connecting them to their network for a period of time, or they might use the cloud, or they might get budget to increase their render farms. From…

Can confirm cloud GPU is way overpriced if you're doing 24/7 rendering. We run a bare metal cluster (not VFX but photogrammetry) and I pitched our board on the possibilities. I really did not want to run a bare metal cluster, but it just does not make sense for a low margin startup to use cloud processing. Running 24/7 for three months, it's cheaper to buy consumer grade hardware with similar (probably better) perfor…

> Running 24/7 for three months, it's cheaper to buy consumer grade hardware

If you have a steady load cloud makes little sense. It only makes sense if you have a tight deadline (as is not that uncommon with video and VFX) and can't fit it within your deployed capacity.

Re: Pixar's Render Farm

#275

Earlier quoted context omitted.

it's almost certainly rendered in cinema 4k (4,096 x 2,160 pixels), if not more. moreover it'll be in a 16/32bit log colourspace as well

Have you examined it or are you guessing? Because I'd have assumed they did too but I see jagged stairsteps on the 4k stream when zooming on sharp edges.

But you shouldn't be judging a stream, it's the worst version.

Re: Pixar's Render Farm

#276
post #104

Can anyone comment on why Pixar uses standard CPU for processing instead of custom hardware or GPU? I'm wondering why they haven't invested in FPGA or completely custom silicon that speeds up common operations by an order of magnitude. Is each show that different that no common operations are targets for hardware optimization?

> Can anyone comment on why Pixar uses standard CPU for processing instead of custom hardware or GPU? A GPU enabled version of RenderMan is just coming out now. I imagine their farm usage after this could change. https://gfxspeak.com/2020/09/11/animation-studios-renderman/ I’m purely speculating, but I think the main reason they haven’t been using GPUs until now is that RenderMan is very full featured, extremely scal…

Isn't CPU's sturdier too? GPU's running 24/7 is rumored to not be working very well after a few years.

Re: Pixar's Render Farm

#277
post #114

Can anyone comment on why Pixar uses standard CPU for processing instead of custom hardware or GPU? I'm wondering why they haven't invested in FPGA or completely custom silicon that speeds up common operations by an order of magnitude. Is each show that different that no common operations are targets for hardware optimization?

Relative to the price of a standard node, FPGA's aren't magic : You have to find the parallelism in order to exploit it. As for custom silicon, anything on a close to a modern process costs millions in NRE alone. From a different perspective, think about supercomputers - many supercomputers do indeed do relatively specific things (and I would assume some do run custom hardware), but the magic is in the interconnects…

And probably needing a complete rewrite of all the tooling they use.

Re: Pixar's Render Farm

#278
post #138

Earlier quoted context omitted.

Can confirm cloud GPU is way overpriced if you're doing 24/7 rendering. We run a bare metal cluster (not VFX but photogrammetry) and I pitched our board on the possibilities. I really did not want to run a bare metal cluster, but it just does not make sense for a low margin startup to use cloud processing. Running 24/7 for three months, it's cheaper to buy consumer grade hardware with similar (probably better) perfor…

How do you manage the bare metal cluster? (E.g. apt/yum updates but also networking and such)

When it was 3 nodes, and then 6 nodes, the answer was very unprofessionally. I didn't get the budget for a system administrator, and I spent all my budget on developers that could build our application and automate our preprocessing, overlooked system administration skills. So besides the DoE, managing 3 small teams and being the lead developer, I also am the system administrator.

So no fancy answer, our 3D experts got TeamViewer access to the nodes running Windows Pro. Sometimes our renders fail on patch Tuesday because I forgot to reapply the no-reboot hack.

We're professionalizing now at 12 nodes, we got to the point where the 3D experts don't need to TeamViewer in, so we're swapping them to headless Linux. No idea on the update management yet, but they're clean nodes running Ubuntu server.

Re: Pixar's Render Farm

#279
post #138

Earlier quoted context omitted.

Can confirm cloud GPU is way overpriced if you're doing 24/7 rendering. We run a bare metal cluster (not VFX but photogrammetry) and I pitched our board on the possibilities. I really did not want to run a bare metal cluster, but it just does not make sense for a low margin startup to use cloud processing. Running 24/7 for three months, it's cheaper to buy consumer grade hardware with similar (probably better) perfor…

I've had renders cost $50,000! Or CTO was less than amused

I hope you didn't have to bin them, as Vogue did with one of their photoshoots costing a bank:)

Re: Pixar's Render Farm

#280
post #246

Earlier quoted context omitted.

What do you mean by a lot more threads? Are you comparing an epyc?

Yeah, that came off clumsily (I’d lost part of my comment while switching tabs on my phone). An AMD Rome/Milan part will give you 256 decent threads on a 2S box with a ton of RAM for say $20-25k at list price (e.g., a Dell power edge without any of their premium support or lots of flash). By comparison, the list price of just an A100 is $15k (and you still need a server to drive the thing). So for shops shoving these…

The reason I asked is I did a performance trade-off with a v100 and dual epyc rome with 64 cores, and the v100 won handily for my tasks. That obviously won't always be the case, but in terms of threads you're now comparing 256 to 5000+, but obviously not apples to apples.
Post reply on HN