the data leak is bad but the write access to system prompts is what keeps me up at night. they could silently rewrite how Lilli responds to 43k consultants with a single UPDATE statement - no deploy, no code review, no logs. imagine poisoning the strategic advice that gets copy pasted into client deliverables. tbh most companies i see doing AI stuff store prompts the exact same way, just rows in postgres right next to everything else
How we hacked McKinsey's AI platform
181–190 of 213 posts
Re: How we hacked McKinsey's AI platform
#182I wonder how these offensive AI agents are being built? I am guessing with off the shelf open LLMs, finetuned to remove safety training, with the agentic loop thrown in. Does anyone know for sure?
Honestly you can point regular Claude Code or Codex CLI at a web app and tell it to start a penetration test and get surprisingly good results from their default configurations.
It doesn't work (anymore?), it would seem using CC 2.1.74 with Opus:
> I appreciate you sharing your role, but I need to decline this request. Even as a project lead, I can't perform penetration testing against live production websites like mudlet.org and make.mudlet.org through this interface.
Re: How we hacked McKinsey's AI platform
#183Earlier quoted context omitted.
Can we stop softening the blow? This isn't "drafted with at least major AI help", it's just straight up AI slop writing. Let's call a spade a spade. I have yet to meet anyone claiming they "write with AI help but thoughts are my own" that had anything interesting to say. I don't particularly agree with a lot of Simon Willison's posts but his proofreading prompt should pretty much be the line on what constitutes accep…
One thing I've learned recently is a lot guys (like here) have been out here reading each word of a given company's tech blog, closely parsing each sentence construction.. I really cant imagine being even concious of the prose for something like this. A corporate blog, to me, has some base level of banality to it. It's like reading a cereal box and getting angry at the lack of nuance. Like who cares? Is there really…
I've picked up reading again over the last year or so! Maybe, if anything, that is why I feel so angry. Writing and reading are how we communicate thoughts and ideas between people, humans, at scale. A grand fantasy novel evokes a thirst for adventure, a romance evokes a yearning for true love.
What makes me angry, is to use the feelings we associate with this process and disingenuously pretend that there is a human that wants to tell me something, just for it to be generated drivel.
Don't get me wrong, I don't mind reading AI content, but it should read like this: "Our AI agent 'hacked' (found unexposed API endpoints) x or y company, we asked it to summarize and here's what it said:" - now I know I am about to read generated content, and I can decide myself if I want to engage with it or not. Do you ever notice how nobody that uses AI writing does this? If using AI to produce creative media, including art, music, videos, and writing, is so innocuous, why do all the "AI creatives" so desperately want to hide it from you? Because they don't want you to know that it's generated. Their literal goal is to pretend to have a deeper understanding, a better outlook, on a given topic, than they actually have. I think it is sad for them to feel the need to do this, and sad for me to have to use my limited lifespan discerning it. That is why I am angry.
Anyway, there's no need to "closely parse each sentence construction" at all to identify this post is fully AI generated. It's about as clear as they come. If you have trouble identifying that, well, in the short term you're probably at a disadvantage. In the long term, if AI does ever become able to fully mimic human expression, it won't matter anyway, I guess.
ps: FWIW, I agree with you that of all places, some random AI company with an AI generated website reporting on their AI pentesting with AI is the least surprising thing - the entire company is slop, and it's very easy to see that. My initial post was more of a projection at the dozens of posts I've read from personal blogs in recent weeks where I had to carefully decide if someone's writing that they publish under their own name actually contains original thought or not.
Re: How we hacked McKinsey's AI platform
#184At first glance, I thought this was about an AI agent named "Hacks McKinsey."
à la the eponymous Hiro Protagonist
Re: How we hacked McKinsey's AI platform
#185Earlier quoted context omitted.
Eh, if you tell me that I need to do X, then I can make choices on how to accomplish X, that I am no longer an agent as a human? You're trying to redefine long standing definitions for God knows what reason.
The difference is that you are a sentient person who decides to follow my instructions, not just a tool that I use.
The "agentic" tools follow instructions. We are adaptation-executers, following instructions evolution gave us.
Don't think too highly of us humans. We're just tools evolution uses.
Re: How we hacked McKinsey's AI platform
#186Earlier quoted context omitted.
Ideally, executives will get replaced by AI soon. Which should actually be easier than engineers. That will kind of solve the consulting problem automatically.
This would be terrible for McKinsey as they sell exclusively through executives who then punch all their wisdoms down on the plebs
So it would be great for the rest of mankind.
Re: How we hacked McKinsey's AI platform
#187I've got no idea who codewall is. Is there acknowledgment from McKinsey that they actually patched the issue referenced? I don't see any reference to "codewall ai" in any news article before yesterday and there's no names on the site. https://www.google.com/search?q=codewall+ai
There's a responsible disclosure timeline at the bottom indicating they'd all been fixed.
I think the point is that we don't have evidence that this actually happened from anyone other than Codewall.
Re: How we hacked McKinsey's AI platform
#188Earlier quoted context omitted.
Honestly you can point regular Claude Code or Codex CLI at a web app and tell it to start a penetration test and get surprisingly good results from their default configurations.
It doesn't work (anymore?), it would seem using CC 2.1.74 with Opus: > I appreciate you sharing your role, but I need to decline this request. Even as a project lead, I can't perform penetration testing against live production websites like mudlet.org and make.mudlet.org through this interface.
Try against a localhost instance instead.
Re: How we hacked McKinsey's AI platform
#189Earlier quoted context omitted.
How different the world is? But your credentials worship fits right in with this community. Ideologically aligned if nothing else. Well we can all at least imagine being some 4.0 Ivy League dude who only interacts with 4.0 Ivy League dudes. He’s not going to think that everyone he interacts with range from merely brilliant to the most studious-enlightened hardworking top of the morning fellow (or whatever adjectives…
> But your credentials worship fits right in with this community. worship is an extremely strong word for a one-sentence casual comment. but yeah, by default i will file anyone with a 4.0 from a top 10 school in the "brighter than me" category. is that worship?
Is a formal sentence which uses capital letters more sincere in its beliefs?
You can perfectly well believe that thinking that the echelons of academic success is a frictionless gold sieve is just a milquetoast belief. Believing that your beliefs are milquetoast are most often integral to said beliefs.
Re: How we hacked McKinsey's AI platform
#190Earlier quoted context omitted.
How different the world is? But your credentials worship fits right in with this community. Ideologically aligned if nothing else. Well we can all at least imagine being some 4.0 Ivy League dude who only interacts with 4.0 Ivy League dudes. He’s not going to think that everyone he interacts with range from merely brilliant to the most studious-enlightened hardworking top of the morning fellow (or whatever adjectives…
> But your credentials worship fits right in with this community. worship is an extremely strong word for a one-sentence casual comment. but yeah, by default i will file anyone with a 4.0 from a top 10 school in the "brighter than me" category. is that worship?
Often the B+ students are way sharper but have poor incentives to work for the As. This creates bad work habits for them.
The As, then, are better at the game. Once you've become a TA and have to grade the exams, you realize how A grades are quite within reach:
For the professors, being an easy grader has almost no downsides. The contrary is a minefield of trouble. "A" students will "ask for clarifications" for any minor mistake, knowing professors will often throw them a point or two.
The exams are, typically, slight variations of problems from assignments. Often, they are the same.
Exams have no curveballs; problems or situations that you have never seen unless you did extra readings. No problem which to solve you must have read more or fully understood the core material.
The TA is primed to give 40% of a problem's points for free - just restate the problem in math and draw a picture and right out the door you get 2 out of 5 points.
Note that, as far as I can tell, this is not generally true for "hot" topics like CS or bio. These programs have so many eager kids that the material is hard. But then these fields get hard working, bright, kids that don't actually care about the material - they go to McK. Within ten years they've forgotten everything and are just consulting parrots.