Live data from Hacker News

Pixar's Render Farm

twitter.com

121–130 of 323 posts

Re: Pixar's Render Farm

#121
post #85

Earlier quoted context omitted.

I'm assuming these 1TiB textures are procedural generated or composites? Where do this large of textures come up?

Can be either. You usually have digital artists creating them. https://en.wikipedia.org/wiki/Texture_artist

Texture artists aren't painting 1 terabyte textures dude.

Re: Pixar's Render Farm

#122

One of the things they mentioned briefly in a little documentary on the making of Soul is that all of the animators work on fairly dumb terminals connected to a back end instance. I can appreciate that working well when people are in the office, but I'm amazed that worked out for them when people moved to work from home. I have trouble getting some of my engineers to have a stable connection stable enough for VS Code…

The entire studio is VDI based (except for the Mac stations, unsure about Windows), utilizing the Teradici PCoIP protocol, 10Zig zero-clients, and (at the time, not sure if they've started testing the graphical agent), Teradici host cards for the workstations.

I was an intern in Pixar systems for 2019 (at Blue Sky now), and we're also using a mix of PCoIP and NoMachine for home users. We finally figured out a quirk with our VPN terminal we sent home with people that was throttling connections, but the experience post-that fix is actually really good. There are a few things that can cause lag (such as moving apps like Chrome/Firefox), but for the most part unless your ISP is introducing problems it's pretty stable. And everyone with a terminal setup has two monitors, either 2*1920x1200 or 1920x1200+2560x1440.

I have a 300Mbps/35Mbps plan (turns into a ~250/35 on VPN) and it's great. We see bandwidth usage ranging from 1Mbps to ~80 on average. The vast majority being sub-20. There are some outliers that end up in mid-100s, but we still need to investigate those.

We did some cross country tests with our sister studio ILM over the summer and was hitting ~70-90ms latency which although not fantastic, was still plenty workable.

Re: Pixar's Render Farm

#123
post #47

For those that can't stand Twitter's UI: https://threadreaderapp.com/thread/1345146328058269696.html

It's very painful to follow a conversation on Twitter. I'm not sure why they think the way they've done things makes sense.

As a reading medium seeing a bunch of tweets strung together is not fantastic as implemented today.

As an authoring medium though, the character constraints force you to write succinct points that keep the reader engaged. You can focus your writing just on one point at a time, committing to them when you tweet, and you can stop anytime. If you're struggling with writing longer form pieces a tweet thread is a great on-ramp to get the outline together, which you can later expand into a post.

As a conversation medium, it's also nice to be able to focus conversation specifically on a particular point, rather than get jumbled together with a bunch of unrelated comments in the comments section at the end of a post.

Re: Pixar's Render Farm

#124

One of the things they mentioned briefly in a little documentary on the making of Soul is that all of the animators work on fairly dumb terminals connected to a back end instance. I can appreciate that working well when people are in the office, but I'm amazed that worked out for them when people moved to work from home. I have trouble getting some of my engineers to have a stable connection stable enough for VS Code…

That is correct. It's pretty common for a technical artist to have a 24-32 core machine, with 128 GB of RAM, and a modern GPU. Not to mention that the entirety of the movie is stored on a NFS and can approach many hundreds of terabytes. When you're talking about that amount of power and data, it makes more sense to connect into the on-site datacenter.

Re: Pixar's Render Farm

#125
post #117
post #54

Earlier quoted context omitted.

Because the expense is not really worth it - even GPU rendering (while around 3/4 x faster than CPU rendering) is memory constrained compared to CPU rendering, and as soon as you try and go out-of-core on the GPU, you're back at CPU speeds, so there's usually no point doing GPU rendering for entire scenes (which can take > 48 GB of RAM for all geometry, accel structures, textures, etc) given the often large memory re…

> Because the expense is not really worth it I disagree with this takeaway. But full disclosure I’m biased: I work on OptiX. There is a reason Pixar and Arnold and Vray and most other major industry renderers are moving to the GPU, because the trends are clear and because it has recently become ‘worth it’. Many renderers are reporting factors of 2-10 for production scale scene rendering. (Here’s a good example: https…

I write a production renderer for a living :)

So I'm well aware of the trade offs. As I mentioned, for lookdev and small scenes, GPUs do make sense currently (if you're willing the pay the penalty of getting code to work on both CPU and GPU, and GPU dev is not exactly trivial in terms of debugging / building compared to CPU dev).

But until GPUs exist with > 64 GB RAM, for rendering large scale scenes, it's just not worth it given the extra burdens (increased development costs, heterogeneous sets of machines in the farm, extra debugging, support), so for high-end scale, we're likely 3/4 years away yet.

Re: Pixar's Render Farm

#126
post #119

Earlier quoted context omitted.

We are way way past that point. It’s time for Twitter to evolve in so many ways.

I cant even remember the last user feature changes Twitter has.

Proper support in the UI for RTs with comments.

But honestly I'd prefer if they spent some fucking time upstaffing their user service and abuse teams. And if they could finally ban Trump and his conspiracy nutcase followers/network.

Re: Pixar's Render Farm

#127

Earlier quoted context omitted.

Nice to have an industry insider perspective on here ;) Can you speak to any competitive advantages a vfx-centric gpu cloud provider may have over commodity AWS? Even the RenderMan XPU looks to be OSL / Intel AVX-512 SIMD based. Thanks! Supercharging Pixar's RenderMan XPU™ with Intel® AVX-512 https://www.youtube.com/watch?v=-WqrP50nvN4

One potential difference is that the input data required to render a single frame of a high end animated or VFX movie might be several hundred gigabytes (even terabytes for heavy water simulations or hair) - caches, textures, geometry, animation & simulation data, scene description. Often times a VFX centric cloud provider will have some robust system in place for uploading and caching out data across the many nodes…

As a specific example, Disney released the data for rendering a single shot from Moana a couple of years ago. You can download it here: https://www.disneyanimation.com/data-sets/?drawer=/resources...

Uncompressed, it’s 93Gb of render data, plus 130Gb of animation data if you want to render the entire shot instead of a single frame.

From what I’ve seen elsewhere, that’s not unusual at all for a modern high end animated scene.

Re: Pixar's Render Farm

#128
post #3

My understanding (I am not an authority) is that for a long time, it has taken Pixar roughly an equal amount of time to render one frame of film. Something on the order of 24 hours. I don’t know what the real units are though (core-hours? machine-hours? simple wall clock?) I am not surprised that they “make the film fit the box”, because managing compute expenditures is such a big deal! (Edit: When I say "simple wall…

Not every place talks about frame rendering times the same. Some talk about the time it takes to render one frame of every pass sequentially, some talk about more about the time of the hero render or the longest dependency chain, since that is the latency to turn around a single frame. Core hours is usually separate because most of the time you want to know if something will be done overnight or if broken frames can be rendered during the day.

24 hours of wall clock time is excessive and the reality is that anything over 2 hours starts to get painful. If you can't render reliably over night, your iterations slow down to molasses and the more iterations you can do the better something will look. These times are usually inflated in articles. I would never accept 24 hours to turn around a typical frame as being necessary. If I saw people working with that, my top priority would be to figure out what is going on, because with zero doubt there would be a huge amount of nonsense under the hood.

Re: Pixar's Render Farm

#129
post #34

Oh man, I wanted this to contain much more details :( Whats the hardware? How much electric energy goes into rendering a frame or a whole movie? How do they provision it (as they keep #cores fixed)? They only talk about cores, do they even use GPUs? What's running on the machines? What did they optimize lately? So many questions! Maybe someone from Pixar's systems department is reading this :)?

Also, renderfarms are usually referred to "in cores", because it's usually heterogeneous hardware networked together over the years. You may have some brand new 96 core 512 GB RAM machines mixed in with some several year old 8 core 32 GB machines. When a technical artist is submitting their work to be rendered on the farm, they often have an idea of how expensive their task will be. They will request a certain number of cores from the farm and a scheduler will go through and try to optimize everyone's requests across the available machines.

Re: Pixar's Render Farm

#130

Earlier quoted context omitted.

Nice to have an industry insider perspective on here ;) Can you speak to any competitive advantages a vfx-centric gpu cloud provider may have over commodity AWS? Even the RenderMan XPU looks to be OSL / Intel AVX-512 SIMD based. Thanks! Supercharging Pixar's RenderMan XPU™ with Intel® AVX-512 https://www.youtube.com/watch?v=-WqrP50nvN4

One potential difference is that the input data required to render a single frame of a high end animated or VFX movie might be several hundred gigabytes (even terabytes for heavy water simulations or hair) - caches, textures, geometry, animation & simulation data, scene description. Often times a VFX centric cloud provider will have some robust system in place for uploading and caching out data across the many nodes…

To re-enforce this, here is some discussion of average machine memory size at Disney and Weta two years ago:

https://twitter.com/yiningkarlli/status/1014418038567796738

Post reply on HN