21–30 of 56 posts
One particular thing to be careful of are core dumps. What I did at a previous shop was remove the passwords as part of a smart gdb script that runs when the core is dumped, before it gets written to a readable location. Writing the script also helped to demonstrate how to extract the passwords in the first place.
Earlier quoted context omitted.
Why is logging everything considered lazy?
for one it's extremely costly, in vcpu , storage , transfer rates. and if you're paying a third-party logger , multiply each by 10x
And an exact match is just part of the problem; if a dev redacts the end and another dev redacts the start, you can still reassemble the secret with enough logs.
I think the big problem is when secrets can be anywhere in a string and you don't control the input (e.g, library stacktraces, HTTP responses, JSON that was stringified). You need to pass the secrets to the logger so it can be redacted, it's heavily dependent on the dev and easy to forget during review. And an exact match is just part of the problem; if a dev redacts the end and another dev redacts the start, you can…
Regex matching on logs is slow but if performed on every node the CPU load is distributed vs. doing this upstream. Configuration management can push the regex rules to all the nodes. This won't help with unknown-unknowns but those can be added quickly to all nodes through configuration management after peer review.
Rsyslog also supports encrypting the log stream so that secret leakage is limited to the sending nodes and the central nodes and it checks a few boxes.
Another thing that helps is limiting to warn and above sent upstream and using an agent on the local nodes to monitor for keywords in the range of info to debug to let someone know to go check the node logs. Less junk on the centralized servers that may have SOC1/SOC2/PCI/FEDRAMP log retention requirements. One can not leak what is not sent in the first place.
I think secrets ending up in the log is an issue but who should have access to view logs of what log should also be an important that is often ignored. This is also scope down the surface area of leakage.
Even if I trust me.
Audits happen. I assume other people will eventually see this bad practice.
Earlier quoted context omitted.
for one it's extremely costly, in vcpu , storage , transfer rates. and if you're paying a third-party logger , multiply each by 10x
That makes it foolish, but I'm not sure if it's lazy.
One particular thing to be careful of are core dumps. What I did at a previous shop was remove the passwords as part of a smart gdb script that runs when the core is dumped, before it gets written to a readable location. Writing the script also helped to demonstrate how to extract the passwords in the first place.
Stack traces, too. I did some work with a heavy Java shop and pretty much everything sensitive ended up in a stack trace at some point.
Earlier quoted context omitted.
for one it's extremely costly, in vcpu , storage , transfer rates. and if you're paying a third-party logger , multiply each by 10x
If you're in a testing environment, where your SIT and UAT are looking to break stuff though, don't you usually want to be able to look to a log of everything?
Secondly, you can't represent the heap & stack well as strings. Concurrent threads and object trees are better debugged with a debugger (e.g. gdb).
The kitchen sink example in particular is one that trips up people. Without knowing the specifics of how a library may deal with failure edge cases, it can catch you off guard (e.g., axios errors including API key headers).
A lot of these problems come from architectures where secrets go over the wire instead of just using signatures/ids. But in cases where you have to use some third party platform, there's often no choice.