Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Keeping secrets out of logs (2024)

allan.reyes.sh

Post reply on HN
Print view View original

11–20 of 56 posts

Re: Keeping secrets out of logs (2024)

#11

Post by jauer »

munchler wrote: post #8

I certainly agree with the desire to keep secrets out of logs, but isn’t the entire log itself also considered to be secret? Even a perfectly sanitized log probably contains lots of data about your production environment that you wouldn’t want to share with adversaries (e.g. peak usage hours).

There’s secret from an adversary and then there’s internal compartmentalization.

You could have 100s of people who have a business need to look at syslog from a router, but approximately nobody who should have access to login creds of administrative users and maybe 10s of people with access to automation role account creds.

Re: Keeping secrets out of logs (2024)

#12

Post by jazzyjackson »

aduwah wrote: post #7

Great article! I will definitely reference it in my upcoming discussions. I had some hard time defending having an EU based o11y stack for our EU based infra. I found it hard to articulate on the spot that there are myriads of places where sensitive/personal data can get in the logs and cause leaks, or make GDPR angry.

Why do I have to know how many letters are in observability? is this some kind of in group signaling?

Re: Keeping secrets out of logs (2024)

#13

Post by dataflow »

As far as run-time exposure prevention goes, I feel like in-band signaling might work better than out-of-band for this problem. Along the lines of the taint checking technique mentioned, you can insert some magic string (say, some recognizable prefix + a randomly generated UUID) into your sensitive strings at the source, that you then strip out at the sink. (Or wrap your secrets in a pair of such magic strings.) Then block or mask any strings containing that magic string from making it into any persisted data, including logs. And it will be easy to identify the points of exposure, since they will be wherever you call your respective seal()/unseal() function or such.

Re: Keeping secrets out of logs (2024)

#15

Post by blkhawk »

oh god - I had that come up in an issue at work just about a month ago. A development system used really simple usernames and passwords since it was just for testing but all the lines with one of those got gobbled up because they had "secrets" in them.

I have very strong opinions on this issue that boils down to. _why are you logging everything you lazy asses_ and _adding all the secrets into another tool just to scan for them in logs just adds another point for them to leak_...

Especially since the ability of lines getting censored even when the secrets were just part of words showed that probably no hashing was involved.

But its a security tool so it stays. I kinda feel like Cassandra but I think I can already predict a major security issue with it or others with the same functionality in the future. its like some goddamn blind spot that software that is to prevent X cannot be vulnerable to X but somehow often is vulnerable because prevention of X and not being vulnerable to X are two separate things somehow.

Re: Keeping secrets out of logs (2024)

#16

Post by pavel_lishin »

blkhawk wrote: post #15

oh god - I had that come up in an issue at work just about a month ago. A development system used really simple usernames and passwords since it was just for testing but all the lines with one of those got gobbled up because they had "secrets" in them. I have very strong opinions on this issue that boils down to. _why are you logging everything you lazy asses_ and _adding all the secrets into another tool just to sca…

Why is logging everything considered lazy?

Re: Keeping secrets out of logs (2024)

#17

Post by mberning »

b0gb wrote: post #5

eazy secrets.forEach(secret => logMessage = logMessage.replaceAll(secret, '**'))

That presumes you know all secrets ahead of time. A risk in and of itself. But from a practical point of view you will never know all secrets, because they are generated constantly in real time.

Re: Keeping secrets out of logs (2024)

#18

Post by mgaunard »

One particular thing to be careful of are core dumps.

What I did at a previous shop was remove the passwords as part of a smart gdb script that runs when the core is dumped, before it gets written to a readable location.

Writing the script also helped to demonstrate how to extract the passwords in the first place.

Re: Keeping secrets out of logs (2024)

#19

Post by aduwah »

jazzyjackson wrote: post #12
aduwah wrote: post #7

Great article! I will definitely reference it in my upcoming discussions. I had some hard time defending having an EU based o11y stack for our EU based infra. I found it hard to articulate on the spot that there are myriads of places where sensitive/personal data can get in the logs and cause leaks, or make GDPR angry.

Why do I have to know how many letters are in observability? is this some kind of in group signaling?

Just wait until you see our secret handshake

Re: Keeping secrets out of logs (2024)

#20

Post by tonymet »

pavel_lishin wrote: post #16
blkhawk wrote: post #15

oh god - I had that come up in an issue at work just about a month ago. A development system used really simple usernames and passwords since it was just for testing but all the lines with one of those got gobbled up because they had "secrets" in them. I have very strong opinions on this issue that boils down to. _why are you logging everything you lazy asses_ and _adding all the secrets into another tool just to sca…

Why is logging everything considered lazy?

for one it's extremely costly, in vcpu , storage , transfer rates. and if you're paying a third-party logger , multiply each by 10x
Post reply on HN