I'm building something that isn't necessarily more secure than vercel, but it is self hosted. I think in the future personal vps family clouds are going to be a lot more common because of these cloud-level attacks and costs.
The Vercel breach: OAuth attack exposes risk in platform environment variables
91–100 of 131 posts
Re: The Vercel breach: OAuth attack exposes risk in platform environment variables
#92[dead]
Re: The Vercel breach: OAuth attack exposes risk in platform environment variables
#93I'm not sure I've seen it mentioned yet that when Vercel rolled out their environment variable UI, there was no "sensitive" option https://github.com/vercel/vercel/discussions/4558#discussion... . There was ~2 years or more until it was introduced https://vercel.com/changelog/sensitive-environment-variables...
A sensitive flag at the UI layer doesn't actually change runtime. Once it's in process.env during a build, any dep that decides to grep it can. The real problem isn't a missing checkbox, it's that we still stuff every secret into one env bag and hand the build tools the whole bag. Cloudflare scoped bindings and Fly already split it up, other platforms are just slower.
Re: The Vercel breach: OAuth attack exposes risk in platform environment variables
#94Some of the details in this report, like the timeline beginning in 2024-2025, haven't been widely reported? Anyone know where these dates are being sourced from? eg, > Late 2024 – Early 2025: Attacker pivots from Context.ai OAuth access to a Vercel employee's Google Workspace account -- CONFIRMED — Rauch statement > Early - mid-2025: Internal Vercel systems accessed; customer environment variable enumeration begins -…
These are all made up and likely hallucinated.
Re: The Vercel breach: OAuth attack exposes risk in platform environment variables
#95I hate environment variables. I hate them so so so much. I can’t think of a single time I would prefer an envvar to a config file.
They’re somewhat necessary when dealing with Docker. But I also hate Docker. So it’s not surprising when one bad design pattern leads to another.
I suppose maybe envvars make sense when dealing with secrets? I’m not sure. I don’t do any webdev. So not sure what’s least bad solution there.
Re: The Vercel breach: OAuth attack exposes risk in platform environment variables
#96Earlier quoted context omitted.
Once you have a session token, which is what you get after you complete the oauth dance, you can issue requests to the API. It is simple as that. The minted token had permission to access the victim's inbox, most likely, which the attacker leveraged to read email and obtain one-time passwords, magic links and other forms of juicy information.
I guess what's unusual is that the scope includes inbox access . IMO it's probably a bad idea to have an LLM/agent managing your email inbox. Even if it's readonly and the LLM behaves perfectly, supply chain attacks have an especially large blast radius (even more so if it's your work email).
It's a bit of a pickle, given that managing your inbox (or at least reading it, classifying and summarizing contents, identifying action items etc.) is one of the most valuable applications of LLMs today, especially if you move beyond software developers having LLMs write code for them.
Re: The Vercel breach: OAuth attack exposes risk in platform environment variables
#97Earlier quoted context omitted.
> with secrets possibly baked into source please don't suggest this. The right way is to have the creds fetched from a vault, which is programmed to release the creds auth-free to your VM (with machine level identify managed by the parent platform) This is how Google Secrets or AWS Vaults work.
This is just another layer of indirection (which isn't bad; it adds to the difficulty of executing a breach). The fundamental problem with encrypted secrets is that at some point you need to access and decrypt them.
I'm not sure that's necessarily a "problem", though it is fundamental to secrets. We wouldn't say that it's a fundamental problem that doors on houses need a key--that's what the key is for--the problem is if the key isn't kept secure from unauthorized actors.
Like, sure, you can go HAM here and use network proxy services to do secret decryption, and only talk from the app to those proxies via short-lived tokens; that's arguably a qualitative shift from app-uses-secret-directly, and it has some real benefits (and costs, namely significant complexity/fragility).
Instead, my favored option is to scope secret use to network locations. If, for example, a given NPM token can only be used for API calls issued from the public IP endpoint of the user's infrastructure, that's a significant added layer of security. People don't agree on whether or not this counts as a "token ACL", but it's certainly ACL-like in its functionality--just controlled by location, rather than identity.
This approach can also be adopted gradually and with less added fragility than the proxy-all-the-things approach: token holders can initially allowlist broad or shared network location ranges, and narrow allowed access sources over time as their networks are improved.
Of course, that's a fantasy. API providers would have to support network-scoped API access credentials, and almost none of them do.
Re: The Vercel breach: OAuth attack exposes risk in platform environment variables
#98Earlier quoted context omitted.
Once you have a session token, which is what you get after you complete the oauth dance, you can issue requests to the API. It is simple as that. The minted token had permission to access the victim's inbox, most likely, which the attacker leveraged to read email and obtain one-time passwords, magic links and other forms of juicy information.
If they had SSO sign in to their admin panel (trusted device checks notwithstanding) the oauth access would be useless. Vercel is understandably trying to shift all the blame on the third party but the fact their admin panel can be accessed with gmail/drive/whatever oauth scopes is irresponsible.
That's a low-leverage place to intervene. Whether or not the internal admin system was directly OAuth linked to Google, by the time the attacker was trying that, they already had a ton of sensitive/valuable info from the employee's Google Workspace account.
If you can only fix one thing (ideally you'd do both, but working in infosec has taught me that you can usually do one thing at most before the breach urgency political capital evaporates), fix the Google token scope/expiry, or fix the environment variable storage system.
Re: The Vercel breach: OAuth attack exposes risk in platform environment variables
#99Earlier quoted context omitted.
> with secrets possibly baked into source please don't suggest this. The right way is to have the creds fetched from a vault, which is programmed to release the creds auth-free to your VM (with machine level identify managed by the parent platform) This is how Google Secrets or AWS Vaults work.
This is just another layer of indirection (which isn't bad; it adds to the difficulty of executing a breach). The fundamental problem with encrypted secrets is that at some point you need to access and decrypt them.
Yep. Then you run into the issue of where to store the secret encryption key.
Security researchers always need to give an answer whenever there's a security incident and the answer can never be "too much centralization risk" even when that is the only reasonable answer. You can't remove centralization risk.
IMO, the future is; every major centralized platform will be insecure in perpetuity and nothing can be done about it.
Re: The Vercel breach: OAuth attack exposes risk in platform environment variables
#100What bites people: rotating a vercel env variable doesn't invalidate old deployments, because previous deploys keep running with the old credential until you redeploy or delete them. So if you rotated your keys after the bulletin but didn't redeploy everything, then the compromised value is still live. Also worth checking your Google Workspace OAuth authorizations. Admin Console > Security > API Controls > Third-part…
Preview deploys are even worse. Every PR spins one up with the same env vars and nobody ever cleans them up. You rotate the key, redeploy prod, and there are still like 200 zombie previews sitting there with the old value.