Live data from Hacker News

Security Certifications Are Causing More Harm Than Good

tacnetsol.com

171–180 of 224 posts

Re: Security Certifications Are Causing More Harm Than Good

#171
post #68

The thing with infosec is that no matter if you're a consultant pen tester or an in-house member of a blue team, a high proficiency in technical writing is required. And few certs demonstrate that the person is a good technical writer. It's not enough to know the answers to multiple choice questions. It's not even enough to know how to exploit things. If you don't understand something well and can discuss it in techn…

It's unlikely that typical OSCP-holder could write a modern buffer overflow exploit, or even judge exploitability of a memory corruption flaw given the source code and a traceback. Equally importantly: memory corruption exploit development and SQL injection are different skills , and most people who do SQL injection don't need proficiency in "buffer overflows". Why is superficial coverage of "buffer overflows" part o…

Totally agree with this. I think security / infosec is just seen as a small niche, so people lump the cryptographers, auditors, SOC analyst, malware analyst, appsec, incident response type people all in one group. It's hard to find someone who can cover all that ground proficiently. Of course it's also been my findings that people who end up in infosec tends to be generalists, but I wonder if that is shifting now that you see more cyber security school initiatives.

Total tangent but, I am absolutely grossed out by "cyber" winning out in the name game. Who let the DoD drive that? Damnit!

Re: Security Certifications Are Causing More Harm Than Good

#172
post #68

The thing with infosec is that no matter if you're a consultant pen tester or an in-house member of a blue team, a high proficiency in technical writing is required. And few certs demonstrate that the person is a good technical writer. It's not enough to know the answers to multiple choice questions. It's not even enough to know how to exploit things. If you don't understand something well and can discuss it in techn…

It's unlikely that typical OSCP-holder could write a modern buffer overflow exploit, or even judge exploitability of a memory corruption flaw given the source code and a traceback. Equally importantly: memory corruption exploit development and SQL injection are different skills , and most people who do SQL injection don't need proficiency in "buffer overflows". Why is superficial coverage of "buffer overflows" part o…

This is why when I was in an infosec bootcamp I begged you to talk to my class and give a dose of reality.

The leet kids, a minority few and the rest naïve, I would bribe while they whittled away at online CTFs and MicroCorruption and irritate them with mediocre questions until they tuned me out. I did not care for tools; approach and mindset are order of magnitudes harder to explain.

I thought you could be a wakeup call had you told them all the certs pro se is a waste in a program that pushed that nonsense. I talk trash of my certs and skills the whole time and they did not get why.

I know you're busy, but I've read your blog and you're preaching to the choir. Starfighter folded, but I would pay for you to test me as a customer and find a mentor to answer my stupid questions that I would pay handsomely for the privilege. I feel I'm not the only one, if you get tired of NCC, that would be amazing!

Re: Security Certifications Are Causing More Harm Than Good

#173

Earlier quoted context omitted.

Don't know if you have taken it in the last year or so since they updated it, but it's pretty tough. You may be able to use a public exploit to elevate your shell once on a box, but getting code execution was the difficult part. One of the challenges involved fuzzing, writing custom buffer overflow exploits, and dealing with weird stack pivots. That only got me about 20% of the way to passing the test. All in 24hrs.…

Describe the overflow exploit you wrote. What was the vulnerability, and what did the exploit look like?

Unfortunately I can't get into too much detail because I had to sign an NDA (to prevent cheating). But the process was similar to when I have found them in the wild: identify the app, install it locally, fuzz various parameters (it was a real application, albeit an old one), find the crash, figure out stack space, figure out bad characters, find the right JMP ESP or equivalent instructions in a loaded library, write shell-code, encode shell-code, slap it all together, hope your hex math doesn't suck, run the exploit. No DEP, ASLR bypass, SEH manipulation, use after free, or heap related work - I learned that on my own.

Their web app challenges were fun too. LFI to code execution, SQL injection, things like that. They have a bunch of network related recon, standard red-teaming stuff.

The OSCE involves ASLR bypass, AV bypass, and using egg hunters.

The big thing about the OSCP, OSCE, OSEE certs is that you actually have to _do_ all of the stuff they teach you. Not a multiple choice or written question in sight. For the test they drop you in a network with vulnerable machines and you have 24, 48, and 72 hours (depending on the cert) to get code execution on each through various techniques. It was challenging, interesting, and satisfying.

Edit - it's worth mentioning that I still find vanilla buffer overflows on projects. These days most thick-client applications that I see are old as hell and are still vulnerable to exploitation techniques from decades ago. So while the skills that the cert makes you prove are cursory and introductory, they are still useful. In any case it's a good starting place for those that want to learn stuff on their own but do better when they are given the push to prove it.

Re: Security Certifications Are Causing More Harm Than Good

#174
post #68

The thing with infosec is that no matter if you're a consultant pen tester or an in-house member of a blue team, a high proficiency in technical writing is required. And few certs demonstrate that the person is a good technical writer. It's not enough to know the answers to multiple choice questions. It's not even enough to know how to exploit things. If you don't understand something well and can discuss it in techn…

It's unlikely that typical OSCP-holder could write a modern buffer overflow exploit, or even judge exploitability of a memory corruption flaw given the source code and a traceback. Equally importantly: memory corruption exploit development and SQL injection are different skills , and most people who do SQL injection don't need proficiency in "buffer overflows". Why is superficial coverage of "buffer overflows" part o…

I totally agree that most web pentesters don't generally need to know how buffer overflow and binary exploitation techniques work but I think an understanding of how low level systems function and how they can be exploited is useful across all security sub-fields.

I don't think Offensive Security is trying to pump out exploitation experts from their entry level cert program. Maybe the higher levels OSCE and OSEE. The intro cert emphasizes breadth over depth. It felt a lot like a cert built around the Exploitation Hackers Handbook.

I think you're thinking of the certs in the wrong light. They are meant to validate baseline knowledge and proficiency, not mastery. If you want to validate mastery you need to look at the persons personal record and work product.

Re: Security Certifications Are Causing More Harm Than Good

#175
post #79

Earlier quoted context omitted.

> There's no reason someone can't have both skills and certifications Of course you're right that it's not impossible. But here's why it happens anyway and why the heuristic of them being roughly mutually exclusive is not insane: 1. There's a certification that's nearly meaningless because it's so easy to obtain without also having the relevant expertise that the certificate is supposed to represent. 2. People who ar…

You've ignored the point of the post you're replying to. You're looking at the credential as a employee signalling tool, not a tool for other parties to satisfy a business need. Your HR department needs avenues to sift through referrals and comparison points. If an individual has the certificate and compares equally with a non-certificate candidate, the first individual has signaled, through the certificate, that he…

> The only time avoiding the certificate entirely is when the signal it provides is negative.

It is possible for the same action send a positive signal when done by some people and a negative signal when done by others. Specifically there can be 'contersignaling' [0], which is basically signaling that you don't need to signal.

[0]: https://kelley.iu.edu/riharbau/cs-randfinal.pdf

Re: Security Certifications Are Causing More Harm Than Good

#176

Earlier quoted context omitted.

Don't know if you have taken it in the last year or so since they updated it, but it's pretty tough. You may be able to use a public exploit to elevate your shell once on a box, but getting code execution was the difficult part. One of the challenges involved fuzzing, writing custom buffer overflow exploits, and dealing with weird stack pivots. That only got me about 20% of the way to passing the test. All in 24hrs.…

Describe the overflow exploit you wrote. What was the vulnerability, and what did the exploit look like?

[deleted]

Re: Security Certifications Are Causing More Harm Than Good

#177
post #68

The thing with infosec is that no matter if you're a consultant pen tester or an in-house member of a blue team, a high proficiency in technical writing is required. And few certs demonstrate that the person is a good technical writer. It's not enough to know the answers to multiple choice questions. It's not even enough to know how to exploit things. If you don't understand something well and can discuss it in techn…

It's unlikely that typical OSCP-holder could write a modern buffer overflow exploit, or even judge exploitability of a memory corruption flaw given the source code and a traceback. Equally importantly: memory corruption exploit development and SQL injection are different skills , and most people who do SQL injection don't need proficiency in "buffer overflows". Why is superficial coverage of "buffer overflows" part o…

What you think about GIAC Platinum, GSE, or whatever it was called? And with a mix of software, networking, and incident handling before the interviews and pentests?

It's the only one I ever thought might be valuable since it combines specific knowledge with some pentest against real systems. Curious what you think as you're more qualified to assess such a thing given all your security evaluations and hiring those people.

Re: Security Certifications Are Causing More Harm Than Good

#178
post #89
post #44

Earlier quoted context omitted.

There's both some truth and some falsehood to the 'certifications don't prove anything' argument: Answer a multiple choice test for an MCSE or whatever? Doesn't prove much. Receive a server that's been wrecked and won't boot, turn it into a load balancing HTTPS server, SMTP server, a bunch of required cron jobs and a boat load more requirements for RHCE? Proves you can do those things. Disclaimer: used to work at Red…

I just recently had someone ask me to take an assessment test for a senior developer position. There's always some silly hoop to jump through, so I thought "why not". Well I got booted out of the test because I hit Ctrl-C to copy something for the first warning, and hit Ctrl-L (muscle memory) for the final revocation of the test. I just thought to myself ... did I just fail an assessment test because I hit Ctrl-L? Th…

I've had several of these in my job search recently as well, with a background in education I'm astonished that anyone thinks they show anything.

Re: Security Certifications Are Causing More Harm Than Good

#179
post #116
post #108

If I'm a competent enough services and web developer wanting to move into infosec, what else could I be doing to get my foot in the door besides collecting certs, as ostensibly shite as they are?

Bug bounties. Andddd that's it, you're done. Find a few in recognizable companies, and jobs will simply come to you. I'm not going to engage in the debate about what certifications should be in the industry, but I'm happy to show which option is most advantageous for your particular needs right now: * Certifications mostly do not teach you anything that you, as a competent web developer, cannot learn from the same fi…

This is the best hot take I've seen on how to deal with certifications in security so far. This is an excellent suggestion.

Re: Security Certifications Are Causing More Harm Than Good

#180

Articles like this one frustrate me. I'm 30, and am essentially starting life over after finishing my military enlistment a couple years ago. all the experience of setting up shops and drafting reports meant nothing with out a degree. So I start working on my degree, and I am absolutely miserable. My love of learning was sucked out of me because I wasn't learning: I was working towards an extra line on my resume. Rig…

Keep your head up. I am a partner at a so called "boutique" firm. If you want to be elite and do the hard things in information security the certifications themselves will not do /much/ for you by themselves. You have to never stop learning and growing. A lot of people in corporate IT that maybe bump into the edges of real assessment work think the certs qualify them and they stop learning and growing and learning how to break software.

It comes down to a very simple concept. Can you make the computer do what you want? Can you find the flaws in its state machine and hack the shit out of it? Yes? Come join an elite firm. No? Go into corporate IT security or keep learning until you can take the raw machine code and make it do what you want.

What does running a bunch of tools have to do with that? Most certs are very tool focused. Some /might/ have you do some stuff that is more interesting and CTF like, but so what. It is still meant for mass certification. If you only study to exploit a buffer overflow or inject SQL you are missing the point (though those are valuable skills).

You need to fundamentally understand. You need to be able to model complex software architectures and understand all the complexity of a modern software architecture and ecosystem quickly. Why quickly? Because it changes quickly. Because there is often so much diversity and complexity for a security practitioner at our level that you have to change architectures seamlessly and at a high (not expert), but very high, level of proficiency. That means you have to write code, play with a diverse amount of modern software and programming languages and constantly be thinking about everything from the security perspective. Learn threat modeling. Learn software. Learn the low level bits of computers and the high level bits.

What does this mean? It means if you know all the command line switches for all the tools on Kali you won't ever get anywhere. You need to write code. You need to understand operating systems like a systems engineer. You need to know what is going on with hardware. Will you use it all every assessment? No. But it will inform and guide your choices and you will have the framework required to understand almost all software and hardware you come across.

We have been working hard on our work sample assessment in our hiring process for the last 9+ months. We have seen folks with an elite level of memory corruption (e.g. guys who find and write exploits for the DoD) experience do very poorly on assessment and we have seen 2nd year college kids get right to the heart of the sample and own it. We see a LOT of people who want to transition into infosec or work at a more hardcore level come in and throw every command on Kali at our work sample. (Amusingly you have to think and assess things, you don't need anything fancier than a hex editor and a programming language or two with their standard library). Does that mean someone good at memory corruption is bad at information security? Maybe. It means their skills are too narrow to assess and secure the typical systems our customers hire us for and we work on, and we work on a lot of important stuff.

So let's be more concrete:

* Get really good at Python or Ruby (Python is what we prefer, but Ruby is okay). Write code every day. Golang is fun and good too.

* Work through all of cryptopals until it hurts, read every paper you can along the way

* Take a couersera course on cryptography Dan Boneh's older one is nice -- you really need to understand the crypto primitives in modern use and how to use them safely, you don't need to know how to implement a side channel resistant AES or ChaCha, but you need to know when someone is screwing up with AES in CBC mode (they almost always are if they are using a crypto primitive)

* Build or contribute to some open source security tools

* Get really good with mitmproxy and or Burp so much stuff now is HTTPS and or WebSockets

* Know your web app LHF

* Read and understand OAUTH (do this later)

* Learn every common authorization model in existence and how authentication and authorization are /actually/ implemented

* Work through Micro corruption CTF, you will understand better how a computer works if you get through /every/ challenge

* Learn threat modeling (Shostack has nice writing about it)

* Find software. Break and threat model software. Find more software.

* Follow and, more importantly, endeavor to understand the work of prominent peoples that talk about BlackHat every year or build software people use (Bernstein, matthew green, and the charlie miller's of the world, understand their methodology first, walk through how they do things more than their results, don't be distracted by results, but the skills and effort they employed to get the results).

That is the basics. Get good at this and you can break most modern software. Then you can specialize. Along the way of doing this you will come across tons of interesting stuff and find places you want to investigate. This is just off the top of my head. This is the really hard thing about being really good... it takes time. You can't just wake up and decide to do this at a high level. Programming takes time. Learning crypto takes time. Learning HTTP takes time. Learning software stacks and modern software architecture takes time. At the end of the day this path is daunting and, like a sieve, it filters out all but the best technologists. Now you can imagine why the author may have taken the sort of down his nose view he did of certifications, because this is an immense and challenging thing.

Step back a bit and assume becoming elite at this is a 5-8 year journey, what do you do in the mean time? Write code every day. Work on only a few things at a time to ensure you can go deep enough and understand it. Do your certs, they give you great exposure to the variety of tech, but never stop at the level a cert gets you to if you want to progress. Figure out what you are enjoying right now and focus on that. You can feasibly get more entry level pen testing and assessment roles in corp security on the backbone of a few certs, getting good at programming and automating things, and going deep on a topic area that really interests you... web app testing is a great starter, but never settle for banging out LHF (Low Hanging Fruit) findings all day, learn how to build web apps, too.

You can also go more of a risk management and policy route. This requires you to have a breadth of knowledge, be deeper with at least a few things, and understand corporate security, but I swear, if you love technology and enjoy deep thinking these roles will suck the life out of you. They are where deep thought often goes to die, drowned by corporate policy. Anyhow, it is getting late. Good luck. Find my company and contact us, we will set you up on our work sample and you can see what it is like.

Post reply on HN