Launch HN: CSPA (YC S18) – SAT for Software Engineers

I scanned through the "General" questions in the practice test linked on the thread, paying particular attention to the "Security" questions.

Not the best.

There are two questions regarding denial-of-service attacks. One of them I found hard to answer (had a plausible answer and a plausible "none of the above"). The second one was clearer but still ambiguous. Weirdly, the ambiguity in both questions stemmed from the use of an HTTP POST login as the possible DOS vector --- why? Why use that confusing example?

The 2FA question I saw had two valid examples of 2FA (push notifications and chip-and-pin) but no "all-of-the-above". I'm curious which of those two the author has demoted.

The phishing question I'm just irritated at the superficiality of; everyone will get the right answer, but the right answer misses the point and the power of phishing attacks, which isn't "websites that look like real websites" but rather the lures (like targeted emails) that get people onto those sites. But, whatever.

The password management advice question asked for a "best" from several subjective answers.

The SMS 2FA question had two valid answers and no matching "these-two-answers" answer. I'm pretty sure the answer that wss being looked for was the social-engineering phone-porting one (which is weird, because the first answer is "SMS 2FA is considered secure by current industry standards", which is certainly true for most reasonable definitions of "industry standards"). But also: that's not even the biggest problem with SMS 2FA.

There was a blockchain question. Who is this for? I'll buy that every working programmer needs to know how a phishing attack works (though perhaps not what distinguishes, in the test author's mind, a DDoS attack from a DoS attack). How many working programmers know how blockchains work?

Similarly: there was a GAN question. Come on. What was the point of that?

Finally: it's a six hour test. You weren't kidding when you called it "the SAT for programmers". It's not an especially pleasant experience (I like the interface, though). This test is a very big ask of candidates, and I think a short ways through the test it becomes clear that there's not much intrinsic merit to it; it's just a hurdle.

They would have to, since this is a general-knowledge programming test.

Was curious as to how tests like these guard against Goodhart's law (If goodheart's law is applicable here at all)?

We've found that that is indeed true. The more senior and knowledgeable you are, the most likely the answer is "it depends", because all the nuances and edge cases you know about.

We try to solve this with the standard "choose the BEST answer". We also do post-mortems and analyze the responses to disqualify or discount bad questions.

These are issues the CSPA Technical Steering Committee, which is responsible for the exam content, will deal with, in collaboration with our research advisor: https://cspa.io/about/team

Our third party proctoring service handles the KYC, as well as the LogMeIn. We did some vetting, and they are used by some well-known MOOCs and universities' online courses. It's a concern of ours as well, which is why we try to offer in-person proctored tests or appointments.