Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

All of Oculus’s Rift headsets have stopped working due to an expired certificate

techcrunch.com

Post reply on HN
Print view View original

271–280 of 390 posts

Re: All of Oculus’s Rift headsets have stopped working due to an expired certificate

#271

Post by zzbzq »

r1ch wrote: post #23

This is not how Windows code signing is supposed to work. Normally you'd get a countersignature from a timestamp server so that the verification process can prove that the certificate was valid at the time of signing. It would appear that Oculus signed their binaries without using a timestamp server, so without a way to verify when signing happened they become invalid as soon as the cert expires.

I don't see enough info in the article to conclude what happened. But I have a hypothesis.

I bet they use certificate pinning.

Process A launches process B and checks against a pinned certificate. This is even more secure than just using the windows code signing stuff.

Problem is, when their cert expired, they were supposed to renew the same cert. Instead, somebody got a new one and signed the build of process B.

The device automatically downloads process B, but then the certificate pin check fails when it tries to launch it.

All the security guides that tell you to do certificate pinning need flashing neon signs explaining this problem. You can't pin certs if you intend to ever change certs.

Re: All of Oculus’s Rift headsets have stopped working due to an expired certificate

#272

Post by binarymidget »

adrianratnapala wrote: post #137
andrewmcwatters wrote: post #96

Earlier quoted context omitted.

I was just reading something here about Cairo and how it's easy to fall into slow code paths with it, and if you happened across falling into a slow code path, somewhere along the line, "you fucked up." When I read the comment I was immediately flabbergasted: no, someone else fucked up. It's not my fault someone wrote software that sets up undocumented traps for me to fall into. Or provided three ways to do something…

The problem in this case is much deeper than their fault / your fault. The problem is that in this industry we do (have to?) lean too much on the power of abstraction. Whether you are writing SQL or graphics code you are constantly told "just express what you want to express directly, and the system is smart enough to do things as efficiently as possible". But that might not be very efficient at all. The people who w…

"...or graphics code you are constantly told "just express what you want to express directly, and the system is smart enough to do things as efficiently as possible"."

Who is saying that? I've worked in video games for 18 years and never heard that. In fact, anyone who said that would get puzzled and suspicious looks. We generally use a very C-like variant of C++.

Re: All of Oculus’s Rift headsets have stopped working due to an expired certificate

#273

Post by mikeash »

yardie wrote: post #238
chrisseaton wrote: post #83

Earlier quoted context omitted.

> It’s not even necessary to test. Once you’ve done it a few times codesigning is a piece of cake. Presumably this is what Oculus thought and so how they got in this mess.

Yes but the test should be, “does this key have a valid timestamp?” not spin up a VM, set the clock 10 years in the future. BTW, most commercial installer progtams will apply a valid timestamp if codesigning is enabled. So to save ~$1000 someone decided the tools built into Visual Studio were good enough. Anyone that ships commercial software that does more than a basic install into C:\program files will know to spen…

The question is, how many other such tests should there be, and what if you’ve missed one?

Re: All of Oculus’s Rift headsets have stopped working due to an expired certificate

#274

Post by agar »

teawrecks wrote: post #268
barkingcat wrote: post #262

Earlier quoted context omitted.

This does not bode well for real VR surgery. Imagine if this were surgery day for someone, and because of an expiring certificate the rift shuts down ...

It's not like a cert is necessary for it to function. A VR headset is basically a monitor you wear on your face. This is their own poor design choice that just ensures they're going to lose business of anyone who needs reliability in their headset.

The "monitor you wear on your face" trope is simply inaccurate, and essentially a misunderstanding of the state of VR today.

I say this not to either criticize you or excuse the mistake by Oculus (they really needed to countersign their cert with a timestamp server), but to educate. These are non-obvious issues to people that don't follow the VR sector.

Monitors work without low-level drivers because their maturity (and lack of innovation) allows the hard stuff to be embedded in the operating system. VR is not at that state; it is emergent, and the capability stacks require additional integration into the OS. Vendors frequently add unique features, and will continue to do so for some time, making standardization difficult.

Even at its simplest level, a VR headset with 6 degrees of freedom is two monitors that must remain in absolute synchronization while also returning positional information to the CPU. This alone is enough to go beyond "standard monitor driver" functionality.

But there's much more. Here is a paste of a comment I made elsewhere:

Oculus (and Steam, via SteamVR) engineers a plethora of low-level code to reduce latency and add features. It's not just a monitor, but a whole set of SDKs, APIs, devices, and drivers.

For the Rift, the hand controllers are wireless input devices refreshing at 1,000Hz; the sensors (to know where you are in the room) are USB devices with tight 60 fps synchronization to LEDs on the headset; there is a custom audio stack with spacialized audio and ambisonic sound; video needs specialized warping to correct lens distortion, interpolate frames, and maintain a 90 fps image, etc.

Not to mention, the system creates a virtual monitor so you can see your 2D desktop while in VR. You can reach out and "grab" a window, rip it from the desktop and position it in your VR world. Pin it, and when you enter a game that window remains and is fully interactive with the Touch controllers emulating a mouse. Maybe you want to play Spotify in the background of a table tennis game, or be able to check a Discord screen while working through a puzzler, or watch YouTube videos while flying your ship in Elite:Dangerous. One guy set up a video feed from his baby monitor so he could watch his kid napping while in VR. This is obviously not a standard feature of the Windows compositor.

All this needs to work across AMD and Nvidia, in Unity, Unreal, or any custom game engine. It's not off-the-shelf driver stuff.

Not to mention, the premise that monitors don't have drivers is also mistaken. They may not be necessary, but they are available[1]. And, the decision to sign kernel drivers is not a poor choice by Oculus, but a mandate from Microsoft for Windows 10 build 1607 and above.[2] A cert is, indeed, necessary to function.

Hope that was informative.

[1] http://www.aocmonitorap.com/my/download_driver.php [2] "Starting with new installations of Windows 10, version 1607, Windows will not load any new kernel mode drivers which are not signed by the Dev Portal." - https://docs.microsoft.com/en-us/windows-hardware/drivers/in...

Re: All of Oculus’s Rift headsets have stopped working due to an expired certificate

#275

Post by Medox »

tudorw wrote: post #192
mattnewport wrote: post #181

Our VR surgical training startup has been working for the last few months towards a big medical conference this week where we're showing multiple training procedures for multiple customers on Oculus Rift, as well as having our own booth. The headsets all stopped working the morning of the conference. Fortunately one of our engineers figured out we could get our demo rigs working by setting the clock back a few days.…

Give the engineer the day off, that's classic side think :)

A day off? That's all? Give that man a raise! Something to look back upon each month. He might have saved the company and even if not, probably a lot of money anyway... it's only fair to give something back.

Re: All of Oculus’s Rift headsets have stopped working due to an expired certificate

#276

Post by JohnTHaller »

Angostura wrote: post #228
JohnTHaller wrote: post #171

Earlier quoted context omitted.

Code signing certificates are different from website certificates. When you use a code signing certificate - when doing it right, anyway - you also loop in a timestamp server. That way, you're signing it with your currently valid certificate and a third party is proving it was signed at a time the certificate is valid. This is so that when your code signing certificate expires in a few months, the binary you signed w…

What makes you think this was a code-signing certificate? The fact that the error involves the inability to reach a server suggests strongly that it was a TLS cert or somesuch

The parent of the comment I replied to was talking about a timestamp server. I was explaining what that meant and why it was using it. I don't know what the actual issue is with the Oculus' handling of certificates and whether it was a code signing certificate or not.

Re: All of Oculus’s Rift headsets have stopped working due to an expired certificate

#277

Post by HeyLaughingBoy »

barkingcat wrote: post #262
mattnewport wrote: post #181

Our VR surgical training startup has been working for the last few months towards a big medical conference this week where we're showing multiple training procedures for multiple customers on Oculus Rift, as well as having our own booth. The headsets all stopped working the morning of the conference. Fortunately one of our engineers figured out we could get our demo rigs working by setting the clock back a few days.…

This does not bode well for real VR surgery. Imagine if this were surgery day for someone, and because of an expiring certificate the rift shuts down ...

As a medical device, I would expect that this possibility would have been caught very early on in one of any number of Failure Analysis meetings and mitigated by the time the device made it to the (FDA) certification process.

Re: All of Oculus’s Rift headsets have stopped working due to an expired certificate

#278

Post by agar »

gmueckl wrote: post #189
zaarn wrote: post #184

Earlier quoted context omitted.

It's still Oculus fault because they didn't use a timestamped signature. A timestamped signature on the binary would have it kept working and that's how MS intents it to work. You can leave it out if you have the desperate longing for having your software break suddenly without reason like Oculus just did.

And that's where MS is at fault: drivers without timestamped signatures should be treated as faulty. This would prevent these errors in the first place.

For what it's worth, I think this is a very sane perspective.

Re: All of Oculus’s Rift headsets have stopped working due to an expired certificate

#279

Post by agar »

OK. The issue arose because the expired certificate wasn't countersigned by a timestamp server.

So many comments agree that (a) security is hard, (b) countersigning with a timestamp server is easy to miss, (c) countersigning makes build processes difficult, and (d) they've done or seen similar things in other apps/companies.

This sounds like a classic UI/UX issue for developers around a literally mandated and mission-critical requirement of the OS.

At the least, MS should provide a validation tool to surface errors or risks before production. Better, signtool.exe should make omissions (like a timeserver) very difficult and make them an override, not a default. Best, they would do both.

I don't agree that the OS should reject non-timestamped signatures as faulty per se (and throw an error), as that puts the burden on the user to understand a developer's mistake. Sometimes running without a timestamp may be desirable - ultimately that's the dev's choice.

It should just be a choice made explicitly.

Re: All of Oculus’s Rift headsets have stopped working due to an expired certificate

#280

Post by lostlogin »

HeyLaughingBoy wrote: post #277
barkingcat wrote: post #262

Earlier quoted context omitted.

This does not bode well for real VR surgery. Imagine if this were surgery day for someone, and because of an expiring certificate the rift shuts down ...

As a medical device, I would expect that this possibility would have been caught very early on in one of any number of Failure Analysis meetings and mitigated by the time the device made it to the (FDA) certification process.

I’m going to assume you haven’t used many bits of medical equipment, because doing it for a job leads me to conclude that the software is more flakey than standard commercial software used day to day. Low sales volumes do not make for budgets high enough to support good debugging and development I guess.
Post reply on HN