Recreating Our Galaxy in a Supercomputer

It was my focus for a long time to achieve perfectly photorealistic rendering. I come from a gamedev background, and I've been fascinated since around age ten about how to get a computer to paint pictures. By 17 I was writing game engines mostly equivalent to quake, and used this portfolio to get into the industry. The next three years were spent getting as close as possible to the bleeding edge of real-time graphics development.

I remember having long debates with a colleague about what high-dynamic range lighting "meant." "If we spin around in our office chair, our brains do not suddenly change the overall brightness of this office. Why should we be programming games to do this? Why is everyone doing things that way?"

My concerns were of a more fundamental nature as well. What is a diffuse texture? A diffuse texture is a well-understood concept in realtime graphics. Anyone with a basic knowledge of shaders should immediately recognize:

  color = lighting * diffuse + ambient

So I turned to research papers. Books. The medical field. Everywhere that was remotely related to possible breakthroughs in photorealistic rendering. Research papers are excellent for assembling techniques, but not results. The books in human vision and color science were more promising, yet most of the industry seemed (and still seems) to pay little attention to them. Compare a book about color perception to, say, http://www.pbrt.org/ and you'll see a stark difference. Flip through the table of contents and you get transformations, shapes, primitives, color and radiometery, sampling, reflection, materials, texture, scattering, light sources, monte carlo integration...

And for what? We know that these techniques simply do not produce computer-generated videos that a human will identify as a real-life image. It's not for lack of processing power. There is a disconnect between the old rules and those that will ultimately result in real-time realism, and you won't find it in that table of contents.

Now, the trouble with writing all of this is that if I knew how to do it, I'd have done it already. It's a life-long search, and it's not so easy to refute an entire industry without being (rightly) dismissed. But if you wish to know what I suspect are the ways forward, it's this: Get a camera. Take photos. Compare these photos to the results of the algorithms you write. Iterate on your algorithms until they are producing results that match something that already captures nature, not our beliefs about how we ought to be able to capture nature. "Just throw in physical models and presto!" has not thus far been true.

You'll notice, for example, that computer graphics in videogames have plateaued. They get more impressive with each generation, but that impressiveness does not get them progressively closer to looking real. Nor should it. A computer game tells a story. The closer it looks to real life, the more restricted the artists are, along with the rest of the design of the game.

So we turn to the movie industry for hope. But it's restricted in exactly the same way. The research papers are all along the lines of new techniques to try, or studies of existing pipelines and how to deal with their complexities. It's not fundamental research.

As someone who has spent his life in pursuit of realism in computer-generated video, my recommendation is this: Read DaVinci's journal. Pay attention to what each page is saying. He had to discover from first principles what makes a painting look real, and why. You'll notice that he spends most of his time talking about human vision and our perceptions of color.

If someone is going to make this development happen, it's not going to come from the game industry, and it won't come from the movie industry. That leaves you. Hopefully this will encourage some of you to pursue this. Once you accept that most of the computer graphics industry isn't actually focused on achieving realism, you'll start to develop your own techniques. My hope is that this will eventually lead to a breakthrough.

It's not so much a novelty as it is pushing the envelope.

The main problem with these kind of simulations is that they're O(N^2). You can't get away with modelling each star, you have to model the interaction of each star with every other star.

And then add in interstellar dust, which has non-neglible mass. And then run the simulation for 8 billion years.

There are naturally some optimizations and short-cuts to be made. :)

Photos, yes. Not video. It's an important distinction because the path to photorealistic video won't pass through impressive-looking still frames. Visual processing of movement is quite different.