Live data from Hacker News

John Carmack: Latency Mitigation Strategies

altdevblogaday.com

31–40 of 106 posts

Re: John Carmack: Latency Mitigation Strategies

#31
It sounds like we would benefit from GPU and display manufacturers providing a set of standard low-level control primitives so that people like Carmack who know what they are doing can really play around with the entire pipeline without having to worry about all the things the cards and the displays are doing behind their backs.

For example, a GPU could have a set of standard settings with full buffering and all the other things that "help them win the framerate wars", but a developer should be able to turn all of that off when needed.

It's the same with displays. LCD manufacturers could, I suppose, allow a modern day "CRT Mode 13h" where you just have scan lines mapped to memory buffers and whatever shows up in those buffers gets turned into a pixel as quickly as possible.

Are there technical challenges preventing this from happening, or is it mainly inertia and lack of need from the current market?

Re: John Carmack: Latency Mitigation Strategies

#32
post #23

Earlier quoted context omitted.

I think it's just a matter of time before the other pieces to the VR puzzle catch up. Omni-directional treadmills are already in existence and are a basic step towards allowing you to walk in a VR environment. Developments in prosthetics with 'feeling' could help pave way towards shoes, gloves or other garments that could provide you with the sensory experiences of walking on different surfaces or picking up in-game…

But at that point you've gone from a games console being something that takes up a small amount of space underneath your TV to something that takes up an entire room. From requiring you to pick up a controller to putting on an entire outfit. I'm just not sure the average consumer is in any way ready/willing for that.

I think the first successful VR you'll see will be closer to an arcade machine than a home console.

Re: John Carmack: Latency Mitigation Strategies

#33

> Conventional computer interfaces are generally not as latency demanding as virtual reality, but sensitive users can tell the difference in mouse response down to the same 20 milliseconds or so, making it worthwhile to apply these techniques even in applications without a VR focus. I wish people would start treating text editors that they are developing this way. As hard real time systems. Don't care much about virt…

Darius Bacon and I have spent some time trying to figure out a buffer data structure that can scale to large files while guaranteeing predictable latency for basic operations like typing a character. (What's "making find call"?)

I think you mean "soft real time", though. A hard real-time system is one which is unusable if it fails to meet a deadline even once; a jet engine control system, say, where failing to meet a deadline could result in engine parts penetrating the fuselage. A "soft real time" system is one where failing to meet a deadline is a failure, but tolerable if infrequent; say, once every million keystrokes, or every 7 days of video play time.

Re: John Carmack: Latency Mitigation Strategies

#34
post #15

> Conventional computer interfaces are generally not as latency demanding as virtual reality, but sensitive users can tell the difference in mouse response down to the same 20 milliseconds or so, making it worthwhile to apply these techniques even in applications without a VR focus. I wish people would start treating text editors that they are developing this way. As hard real time systems. Don't care much about virt…

Would you be willing to pay a recurring fee for using an editor? I have toyed with writing an editor at multiple points but have backed away because I think in the grand scheme of things its a one time sale to a pretty small market (The number of people who care enough about text editing that they would pay for a good editor when they see one)

I might be willing to pay. What would your editor offer that the free, powerful, well-established editors out there do not?

Re: John Carmack: Latency Mitigation Strategies

#35
post #23

Earlier quoted context omitted.

But at that point you've gone from a games console being something that takes up a small amount of space underneath your TV to something that takes up an entire room. From requiring you to pick up a controller to putting on an entire outfit. I'm just not sure the average consumer is in any way ready/willing for that.

I think the first successful VR you'll see will be closer to an arcade machine than a home console.

Agreed. But that's at odds with the OPs assertion that " head-mounted-display virtual reality is a huge missed opportunity for next-gen consoles."

Re: John Carmack: Latency Mitigation Strategies

#36

> Conventional computer interfaces are generally not as latency demanding as virtual reality, but sensitive users can tell the difference in mouse response down to the same 20 milliseconds or so, making it worthwhile to apply these techniques even in applications without a VR focus. I wish people would start treating text editors that they are developing this way. As hard real time systems. Don't care much about virt…

> I wish people would start treating text editors that they are developing this way.

IDEs. Most of the things people do in modern IDEs were done in Smalltalk (albeit, with uglier interfaces and some manual steps) over a decade back. One of the differences, though, is that these things were mostly highly responsive, because they were bloat-free. (Can't speak for IBM VisualAge. Also, one could rightly complain that they were feature-poor in their stock configurations.) This is in stark contrast to other IDEs and app server frameworks, which would take forever to start up, or forever to restart, or forever to complete an operation. One example, Extract Method, was always instantaneous. It's still lugubrious even in modern IDEs like XCode4.5.

The same rules that apply to Web Apps also apply to programmer's tools. Delays are yuck. Responsiveness is yum!

I'm also a musician, and from what I've seen playing around in Audacity, 5 milliseconds is definitely inaudible, even to some of the most talented, golden-eared musicians I know. I suspect that 20 milliseconds is going to start to bug some people, though they won't be able to say why.

Also, awareness of subtle modulations of very small variations in rhythm and how they affect the feel of music seems to be an indicator of intelligence and aesthetic awareness. So often, the difference between blah and great involves things that are subtle and hard to put your finger on.

Re: John Carmack: Latency Mitigation Strategies

#37
> Updating the imagery in a head mounted display (HMD) based on a head tracking sensor is a subtly different challenge than most human / computer interactions.

Doesn't the military already have this solved for head mounted displays for attack helicopters and 4++ generation fighter jets? Heck, they have augmented reality displays for that matter.

EDIT: Many of these problems could be solved by putting an entire purpose-built gaming rig in the headset. There are nicely capable chips for mobile devices with lots of power in the GPU. Design such a system from the ground-up for low latency. Accelerometer and head-tracking inputs would be low-level interrupts, for example.

I wonder if this is how the military contractors solved this?

Re: John Carmack: Latency Mitigation Strategies

#38

It sounds like we would benefit from GPU and display manufacturers providing a set of standard low-level control primitives so that people like Carmack who know what they are doing can really play around with the entire pipeline without having to worry about all the things the cards and the displays are doing behind their backs. For example, a GPU could have a set of standard settings with full buffering and all the…

"a set of standard low-level control primitives ... Are there technical challenges preventing this from happening, or is it mainly inertia and lack of need from the current market?"

It's easier to iterate faster without having to deal with a standardization committee. See OpenGL history for some context.

Re: John Carmack: Latency Mitigation Strategies

#39
post #3

I feel stupid.

The view update part was the most confusing, but I tried to understand it this way:

Imagine you had a magical game engine that rendered the entire world perfectly accurately for every point in space and direction a viewer could possibly be looking at. All you had to do was say:

    RenderGameFrameForEveryPossiblePoint();

    ... // Who knows how much time

    viewerPosition = QuicklyGetViewerPosition();

    TellTheGPUToShowWorldAccordingTo(viewerPosition);
Well then, you could postpone calling those last two functions until the absolute last minute. This way you have very little or no movement of the viewer's head between when you read their position and when you show the view for that position.

But naturally, RenderGameFrameForEveryPossiblePoint() is slightly out of bounds of current technology. A lot of what Carmack was discussing, as I understood it, was simulating this effect as closely as possible. The way to do that, it seems, is:

   StartRenderingGameFrameAccordingTo(lastViewerPosition);
   
   ... // Stuff

   viewerPosition = QuicklyGetViewerPosition();

   viewMatrix = ComputeViewMatrixDelta(lastViewerPosition, viewerPosition);

   FinalizeGPUFrameRenderWithNewViewMatrix(viewMatrix);
That final bit is just a perspective transformation of a bunch of rendering that was already computed and given to the GPU. But if the viewer moves too quickly, you can easily move somewhere in the world that wasn't actually rendered, or your perspective could shift such that and object that was once occluded is now visible, or vice versa. It seems a lot of the complexity is there.

The last thing he talked about, time warping, seems to be a similar thing only it's scanline by scanline. So in effect you're saying "hey, video card and display, I know you're going to force me to draw a whole frame at once, so I'm going to give you a frame where each scanline gets rendered a little bit into the future according to where the player is moving."

The effect on a monitor would probably look like a forward shear, but on an HMD (if done correctly), it would correct for the natural shear caused by having to "freeze frame" the viewer's perspective for one entire frame instead of just a scanline.

Some of this may be woefully incorrect, but it was how I explained it to myself. Please correct anything that's wrong or overly simplified.

Re: John Carmack: Latency Mitigation Strategies

#40

It sounds like we would benefit from GPU and display manufacturers providing a set of standard low-level control primitives so that people like Carmack who know what they are doing can really play around with the entire pipeline without having to worry about all the things the cards and the displays are doing behind their backs. For example, a GPU could have a set of standard settings with full buffering and all the…

Regarding your last point: it's getting better. Modern TVs have "game mode", where latency is minimized. The latest LCD panels use embedded Display Port, eDP. The latest eDP standard introduces a frame buffer on the LCD itself, where you can just write deltas and tell it to "swap". I think this is quite similar to what you're proposing, in concept.
Post reply on HN