Earlier quoted context omitted.
The 4k streaming copy of the film has stairsteps though. Like it's been upsampled. I'm sure their software can render at 4k but they choose not to for whatever reason.
are you sure that's not just an artifact of it being streamed?
Pixar's Render Farm
271–280 of 323 posts
Re: Pixar's Render Farm
#272Earlier quoted context omitted.
Because the expense is not really worth it - even GPU rendering (while around 3/4 x faster than CPU rendering) is memory constrained compared to CPU rendering, and as soon as you try and go out-of-core on the GPU, you're back at CPU speeds, so there's usually no point doing GPU rendering for entire scenes (which can take > 48 GB of RAM for all geometry, accel structures, textures, etc) given the often large memory re…
> while around 3/4 x faster than CPU rendering My understanding is that for neural networks, the speedup is much more than 4x. Does anyone know why there's such a difference?
OTOH, Vfx rendering involves a varying scene with moving light sources, cameras, objects, textures, and physics. Much more dynamic interactions. This is a gross simplification but I hope it helps.
Re: Pixar's Render Farm
#273I wrote about my experiences in more detail a couple of months ago:
https://blog.markjgsmith.com/2020/11/24/what-its-like-workin...
There’s also some render farm setup stuff in my portfolio on the blog if you’re interested.
The hardware side of things was incredible. So much expensive kit. There’s a lot of great hardware related comments in the thread, so I won’t go over that.
The thing that I found really cool was how all the software systems were setup for so many artists to collaborate. Though we didn’t talk about agile methodologies (it was back in 2003/2004), the ways we worked had a lot of similarity to a classic agile / scrum setup, with daily standups, though they were called ‘dailies’, and there was no notion of sprints (we were always sprinting!) or sprint planning, but similar planning was done by producers and department heads, instead of user stories / features, people worked on ‘shots’. At the digital intermediate place the unit of work tended to be reels since we were scanning and printing entire reels, combining all the shots that the vfx houses completed.
Everyone used version control, though it was subversion, I don’t think git was that popular back then. Artists worked on their shots, checking in their project files rather than rendered files, we could always re-render a shot if necessary. There were also cli tools to submit finished rendered files, which were automatically organised into a standard folder structure on shared storage. The artists Linux shells would load environment variables from a dB, and their applications (Shake/Maya/Houdini etc) would load these transparently, so they never had to worry about where things were stored. That was all automatic as long as they knew which shot they were working on.
It was a great place to learn about technology and collaborating on digital production at scale.
I’ve always thought there were a lot of setups and tools that could be applied to software development at scale. I’d love to work on that sort of project. Hint: I am available for hire :)
Re: Pixar's Render Farm
#274Earlier quoted context omitted.
Not Pixar specifically but Modern VFX and Animation studios usually have a bare metal render farm, they usually are pretty beefy -- think at least 24 cores / 128 GB of RAM per node. Usually in crunch time if there's not enough nodes in the render farm they might rent nodes connecting them to their network for a period of time, or they might use the cloud, or they might get budget to increase their render farms. From…
Can confirm cloud GPU is way overpriced if you're doing 24/7 rendering. We run a bare metal cluster (not VFX but photogrammetry) and I pitched our board on the possibilities. I really did not want to run a bare metal cluster, but it just does not make sense for a low margin startup to use cloud processing. Running 24/7 for three months, it's cheaper to buy consumer grade hardware with similar (probably better) perfor…
If you have a steady load cloud makes little sense. It only makes sense if you have a tight deadline (as is not that uncommon with video and VFX) and can't fit it within your deployed capacity.
Re: Pixar's Render Farm
#275Earlier quoted context omitted.
it's almost certainly rendered in cinema 4k (4,096 x 2,160 pixels), if not more. moreover it'll be in a 16/32bit log colourspace as well
Have you examined it or are you guessing? Because I'd have assumed they did too but I see jagged stairsteps on the 4k stream when zooming on sharp edges.
Re: Pixar's Render Farm
#276Can anyone comment on why Pixar uses standard CPU for processing instead of custom hardware or GPU? I'm wondering why they haven't invested in FPGA or completely custom silicon that speeds up common operations by an order of magnitude. Is each show that different that no common operations are targets for hardware optimization?
> Can anyone comment on why Pixar uses standard CPU for processing instead of custom hardware or GPU? A GPU enabled version of RenderMan is just coming out now. I imagine their farm usage after this could change. https://gfxspeak.com/2020/09/11/animation-studios-renderman/ I’m purely speculating, but I think the main reason they haven’t been using GPUs until now is that RenderMan is very full featured, extremely scal…
Re: Pixar's Render Farm
#277Can anyone comment on why Pixar uses standard CPU for processing instead of custom hardware or GPU? I'm wondering why they haven't invested in FPGA or completely custom silicon that speeds up common operations by an order of magnitude. Is each show that different that no common operations are targets for hardware optimization?
Relative to the price of a standard node, FPGA's aren't magic : You have to find the parallelism in order to exploit it. As for custom silicon, anything on a close to a modern process costs millions in NRE alone. From a different perspective, think about supercomputers - many supercomputers do indeed do relatively specific things (and I would assume some do run custom hardware), but the magic is in the interconnects…
Re: Pixar's Render Farm
#278Earlier quoted context omitted.
Can confirm cloud GPU is way overpriced if you're doing 24/7 rendering. We run a bare metal cluster (not VFX but photogrammetry) and I pitched our board on the possibilities. I really did not want to run a bare metal cluster, but it just does not make sense for a low margin startup to use cloud processing. Running 24/7 for three months, it's cheaper to buy consumer grade hardware with similar (probably better) perfor…
How do you manage the bare metal cluster? (E.g. apt/yum updates but also networking and such)
So no fancy answer, our 3D experts got TeamViewer access to the nodes running Windows Pro. Sometimes our renders fail on patch Tuesday because I forgot to reapply the no-reboot hack.
We're professionalizing now at 12 nodes, we got to the point where the 3D experts don't need to TeamViewer in, so we're swapping them to headless Linux. No idea on the update management yet, but they're clean nodes running Ubuntu server.
Re: Pixar's Render Farm
#279Earlier quoted context omitted.
Can confirm cloud GPU is way overpriced if you're doing 24/7 rendering. We run a bare metal cluster (not VFX but photogrammetry) and I pitched our board on the possibilities. I really did not want to run a bare metal cluster, but it just does not make sense for a low margin startup to use cloud processing. Running 24/7 for three months, it's cheaper to buy consumer grade hardware with similar (probably better) perfor…
I've had renders cost $50,000! Or CTO was less than amused
Re: Pixar's Render Farm
#280Earlier quoted context omitted.
What do you mean by a lot more threads? Are you comparing an epyc?
Yeah, that came off clumsily (I’d lost part of my comment while switching tabs on my phone). An AMD Rome/Milan part will give you 256 decent threads on a 2S box with a ton of RAM for say $20-25k at list price (e.g., a Dell power edge without any of their premium support or lots of flash). By comparison, the list price of just an A100 is $15k (and you still need a server to drive the thing). So for shops shoving these…