Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Keeping secrets out of logs (2024)

allan.reyes.sh

Post reply on HN
Print view View original

31–40 of 56 posts

Re: Keeping secrets out of logs (2024)

#31

Post by halffullbrain »

aduwah wrote: post #7

Great article! I will definitely reference it in my upcoming discussions. I had some hard time defending having an EU based o11y stack for our EU based infra. I found it hard to articulate on the spot that there are myriads of places where sensitive/personal data can get in the logs and cause leaks, or make GDPR angry.

I read the piece expecting precisely that; How to keep PII out of logs, which require a lot of adamant snipers with a lot of lead bullets. Passwords: Handled by IAM services. Tokens: Application frameworks which not to divulge. But Brian's phone number stashed in an innocuous case metadata field. Gaah!

Some of the same techniques apply, like using domain primitives, but some PII (like names and addresses) is eventually templated into flatter (text) values, and processed by other layers which do not recognize 'brands' as suggested.

Data scanners: Regexes are fine for SSNs and the like, but to be really effective, one would need a full-on Named Entity Recognition in the pipeline, perhaps just as a canary. (Wait, that might actually work?)

Dataflow analysis and control applies in a BIG way, e.g. separating an audit log for forensics, where you really NEED the PII, from a technical log which the SREs can dig into without being suspected of stealing sensitive info. Start there.

Re: Keeping secrets out of logs (2024)

#32

Post by debarshri »

stretchwithme wrote: post #26
debarshri wrote: post #14

I think secrets ending up in the log is an issue but who should have access to view logs of what log should also be an important that is often ignored. This is also scope down the surface area of leakage.

A user's password is something I shouldn't see in a log, even if I'm in control of what gets logged and frequently access them to do my job. Even if I trust me. Audits happen. I assume other people will eventually see this bad practice.

Audits and bad practice are second-order things.

My argument is that generally everyone has access to all the logs. If you restrict the access and add guardrails around it, you can minimize the surface area and also ways it can be leaked out.

If you take a defensive approach towards, you have to assume that some secret is getting logged somewhere. The goal then becomes a way to reduce the surface area or blast radius of this possible leakage.

Re: Keeping secrets out of logs (2024)

#33

Post by pluto_modadic »

munchler wrote: post #8

I certainly agree with the desire to keep secrets out of logs, but isn’t the entire log itself also considered to be secret? Even a perfectly sanitized log probably contains lots of data about your production environment that you wouldn’t want to share with adversaries (e.g. peak usage hours).

PII is different from proprietary info. customer's email? PII. mask it. your code's stack trace? proprietary info. employees can see that to troubleshoot.

Re: Keeping secrets out of logs (2024)

#34

Post by pluto_modadic »

b0gb wrote: post #5

eazy secrets.forEach(secret => logMessage = logMessage.replaceAll(secret, '**'))

I've known users to type passwords in the username field. you implicitly do NOT know all secrets (e.g., a password is hashed).

secrets can also churn, so even if you did your example would require something besides an in-memory array.

and, the final point: what if your secret masking code fails on an exception, too ;)

Re: Keeping secrets out of logs (2024)

#35

Post by micksmix »

Loved this “lead bullets” framing, especially the parts on taint checking, scanners, and pre-processing/sampling logs. One practical add-on to the "Sensitive data scanners" section is verification: can you tell which candidates are actually live creds?

We’ve been working on an open source tool, Kingfisher, that pairs fast detection (Hyperscan + Tree-Sitter) with live validation for a bunch of providers (cloud + common SaaS) so you can down-rank false positives and focus on the secrets that really matter. It plugs in at the chokepoints this post suggests: CI, repo/org sweeps, and sampled log archives (stdin/S3) after a Vector/rsyslog hop.

Examples:

  kingfisher scan /path/to/app.log --only-valid
  kingfisher scan --s3-bucket my-logs --s3-prefix prod/2025/09/
Baselines help keep noise down over time.

Repo: https://github.com/mongodb/kingfisher (Apache-2.0)

Disclosure: I help maintain Kingfisher.

Re: Keeping secrets out of logs (2024)

#36

Post by HelloNurse »

dataflow wrote: post #13

As far as run-time exposure prevention goes, I feel like in-band signaling might work better than out-of-band for this problem. Along the lines of the taint checking technique mentioned, you can insert some magic string (say, some recognizable prefix + a randomly generated UUID) into your sensitive strings at the source, that you then strip out at the sink. (Or wrap your secrets in a pair of such magic strings.) Then…

Can you elaborate on the situations and reasons that would make this approach appropriate?

At first sight it seems a complicated and inferior approximation of techniques from the article: not automatically single use, not statically checked, somewhat error prone for proper secret usage, not really preventing well-intentioned idiots from accidentally extracting, "laundering" and leaking the secret, removing secrets from logs at a dangerously late stage with some chance of leaks.

Re: Keeping secrets out of logs (2024)

#37

Post by petesergeant »

> If you shift from “any string can be a secret” to “secrets are secrets”, it makes things a lot easier to reason about and protect.

> const secret = new Secret("...")

one of those things that's obvious in retrospect. That's a cute trick I'll definitely be stealing.

Re: Keeping secrets out of logs (2024)

#38

Post by jiggawatts »

petesergeant wrote: post #37

> If you shift from “any string can be a secret” to “secrets are secrets”, it makes things a lot easier to reason about and protect. > const secret = new Secret("...") one of those things that's obvious in retrospect. That's a cute trick I'll definitely be stealing.

.NET has SecureString: https://learn.microsoft.com/en-us/dotnet/api/system.security...

Which reminds me of why I hate tiny standard libraries as seen in JavaScript: features like SecureString work only if they're used pervasively. It has to be in the std lib and it has to be used everywhere so that you almost never have to unwrap them. It's critical that credentials are converted to SecureString as soon as possible and that they stay as SecureString values until the last possible instant when they're passed to some external API call deep inside even a third-party a library.

Re: Keeping secrets out of logs (2024)

#40

Post by jaspervdj »

debarshri wrote: post #32
stretchwithme wrote: post #26

Earlier quoted context omitted.

A user's password is something I shouldn't see in a log, even if I'm in control of what gets logged and frequently access them to do my job. Even if I trust me. Audits happen. I assume other people will eventually see this bad practice.

Audits and bad practice are second-order things. My argument is that generally everyone has access to all the logs. If you restrict the access and add guardrails around it, you can minimize the surface area and also ways it can be leaked out. If you take a defensive approach towards, you have to assume that some secret is getting logged somewhere. The goal then becomes a way to reduce the surface area or blast radius…

Limiting access helps, but if you are storing the logs on a 3rd party (e.g. DataDog, CloudWatch), you will still need to assume it can leak through that 3rd party and start rotating.
Post reply on HN