Earlier quoted context omitted.
I've had renders cost $50,000! Or CTO was less than amused
I hope you didn't have to bin them, as Vogue did with one of their photoshoots costing a bank:)
Sounds interesting: reference?
Pixar's Render Farm
281–290 of 323 posts
Re: Pixar's Render Farm
#282Earlier quoted context omitted.
Everywhere I worked has been traditional NFS, and I've seen more than 3 times the figure you quoted working well. Usually you have different mountpoints/vfs`s in different servers for different kinds of files.
Interesting, maybe the scientific computations we are doing are more I/O intense than render applications? How do studios manage disaster recovery? What happens when a multi petabyte NFS server keels over? Are there tape drive backups? It seems risky to have a such a critical system serviced by only a single node.
At Weta we divided up the NFS servers into "src" and "dat" - "src" was everything made by artists, and "dat" was the output from the renderwall. We backed up "src" every night, but "dat" was never backed up. Every once in a while there would be some mass deletion event but it was always faster to re-render the lost data than to restore from backups.
Also none of the high end commercial filers are single node - they're all clusters of varying sizes.
Re: Pixar's Render Farm
#283Earlier quoted context omitted.
Can confirm cloud GPU is way overpriced if you're doing 24/7 rendering. We run a bare metal cluster (not VFX but photogrammetry) and I pitched our board on the possibilities. I really did not want to run a bare metal cluster, but it just does not make sense for a low margin startup to use cloud processing. Running 24/7 for three months, it's cheaper to buy consumer grade hardware with similar (probably better) perfor…
How do you manage the bare metal cluster? (E.g. apt/yum updates but also networking and such)
Network solutions highly depends on the physical infrastructure, but for setup maintenance, you can often see SaltStack.
Re: Pixar's Render Farm
#284Oh man, I wanted this to contain much more details :( Whats the hardware? How much electric energy goes into rendering a frame or a whole movie? How do they provision it (as they keep #cores fixed)? They only talk about cores, do they even use GPUs? What's running on the machines? What did they optimize lately? So many questions! Maybe someone from Pixar's systems department is reading this :)?
> They only talk about cores, do they even use GPUs?
From my experience (animated movies) GPU is still very experimental because of how limited it can scale, and definitely not use in render farm.
And I'm not even talking about the cost.
GPU rendering demos focus on speed, but of the biggest problem with full feature is flexibility. The more complex your image is, the more problems/artifacts you will "create" on it. Your render time can be x100 faster, if you need to spend two days to fix a problem for each shot, the quality VS speed ratio completely fall over.
Everything get easier, slowly, so maybe one day we will have a 100% GPU farm on big budget projects, but for now, CPU is the most predictable way to manage large scale rendering for both sides (sysadmin/artists).
Re: Pixar's Render Farm
#285Earlier quoted context omitted.
Because the expense is not really worth it - even GPU rendering (while around 3/4 x faster than CPU rendering) is memory constrained compared to CPU rendering, and as soon as you try and go out-of-core on the GPU, you're back at CPU speeds, so there's usually no point doing GPU rendering for entire scenes (which can take > 48 GB of RAM for all geometry, accel structures, textures, etc) given the often large memory re…
> Because the expense is not really worth it I disagree with this takeaway. But full disclosure I’m biased: I work on OptiX. There is a reason Pixar and Arnold and Vray and most other major industry renderers are moving to the GPU, because the trends are clear and because it has recently become ‘worth it’. Many renderers are reporting factors of 2-10 for production scale scene rendering. (Here’s a good example: https…
> There is a reason Pixar and Arnold and Vray and most other major industry renderers are moving to the GPU
The reason is that those softwares need to be sold to many, and a big part of studios are doing advertise and series. GPU rendering is perfect for them as they don't need/can't afford large scale render farms.
About your example, that not honest. It's full of instances and perfect use case for a "Wow" effect but it's not a production shot. Doing a production shot required complexity management on the long run, even for CPU rendering. On this side, GPU is more "constrained" than CPU, management is even more complex.
Re: Pixar's Render Farm
#286It's good to know they care about optimization. I had the assumption that all CGI is a rather wasteful practice where you just throw more hardware at the problem.
> I had the assumption that all CGI is a rather wasteful practice where you just throw more hardware at the problem.
Hardware can be overloaded quickly if you don't care about it. You still need some engineering to keep everything under control.
I suspect this assumption come from the fact CGI have a lot of different things to render so you try to get the hardware that "support" by and large the problems you will have: You can't focus optimization on every single problem, but you can optimize the 90% use case, so the 10% use case can represent 90% of artist time and keep deadline safe.
Re: Pixar's Render Farm
#287Earlier quoted context omitted.
Texture artists aren't painting 1 terabyte textures dude.
The largest texture sets are heading towards 1TB in size, or at least they were when I was last involved in production support. I saw Mari projects north of 650gb and that was 5 years ago. Disclaimer : I wrote Mari, the vfx industry standard painting system. Note though these are not single 1TB textures, they’re multiple sets of textures, plus all of the layers that constitute them. Some large robots In particular ha…
I think we both realize that it's a bit silly to have so much data in textures that you have 100x the pixel data of a 5 second shot at 4k with 32 bit float rgb. 650GB of textures would mean that even with 10gb ethernet (which I'm not sure is common yet) you would wait at least 12 minutes just for the textures to get to the computer before rendering could start and rendering 100 frames at a time would mean 100GB/s from a file server for a single shot. Even a single copy of the textures to freeze an iteration would be thousands in expensive disk space.
I know it doesn't makes sense to tell your clients that what they are doing is nonsense, but if I saw something like that going on, the first thing I would do is chase down why it happened. Massive waste like that is extremely problematic while needing to make a sharper texture for some tiny piece that gets close to the camera is not a big deal.
Re: Pixar's Render Farm
#288It's good to know they care about optimization. I had the assumption that all CGI is a rather wasteful practice where you just throw more hardware at the problem.
CGI is heavily about optimization. I recommend checking out SIGGRAPH ACM papers , and there's a great collection by them on production renderers. Every second spent rendering or processing, is time an artist is not working on a shot. Any savings in optimizations add up to incredible cost savings.
About this: One SIGGRAPH focused was about Multithreading for VFX. The result was so valuable they made a book.
I reviewed this book (2014): https://www.fevrierdorian.com/blog/post/2014/08/24/Multithre...
Re: Pixar's Render Farm
#289I like the picture of the 100+ SPARCstation render farm for the first Toy Story https://mobile.twitter.com/benedictevans/status/766822192197...
This reminds me of one of the first FreeBSD press releases. [0] > FreeBSD Used to Generate Spectacular Special Effects For the then upcoming movie The Matrix > Manex Visual Effects used 32 Dell Precision 410 Dual P-II/450 Processor systems running FreeBSD as the core CG Render Farm. [0]: https://www.freebsd.org/news/press-rel-1.html edit: added that it was for The Matrix.
Wow. I assume for Rendering Farm it is all Linux now?
Re: Pixar's Render Farm
#290Earlier quoted context omitted.
Yes, but the work units (frames) are large enough that I'm still surprised. Maybe they're not as parallelizable as I'd expect. E.g. if there's serial work to be done by reusing scene layout algorithms between frames.
A scene will have many thousands of assets (trees, cars, people, etc) each one will have the geo, which could be in the milllions of polygons (although they use sub-ds) each "polygon" could have a 16k texture on it. You're pulling TBs of textures and other assets in each frame.
Hmm, yes I see. TBs? Interesting. I'd like to hear a talk about these things.
Naively I would expect that (as is the case for my MUCH smaller scale system) that I can compensate for network/disk-bound and non-multithreaded stages by merely running two concurrent frames.
On a larger scale I would expect to be able to estimate RAM-cheap frames, and always have one of them running per machine, but at SCHED_IDLE priority, so that they only get CPU when the "main" frame is blocked on disk or network, or a non-parallelizable stage. By starving one frame of CPU, it's much more likely that it'll need CPU the short intervals when it's allowed to get it.