Saving millions on logging: Finding relevant savings

Post by SOLAR_FIELDS » Sat, Jan 28, 2023, 10:43 AM UTC

coredog64 wrote: post #4

Denzel wrote: post #3

Earlier quoted context omitted.

Yes, shipping computations instead of data is a reasonable design goal. Your proposed system only works when the predicate is independent across all logs though, correct? If you have to correlate or join your logs to anything, then this model becomes more complex. Not to mention, you're adding an additional performance tax to your prod machines which could be more costly than shipping logs to a centralized store. (A…

I once worked on a system where we were told to keep 18 months of debug logs (something that would have cost about $2k/month). When we pushed back and asked why, the answer was that occasionally (every month or every other month) there would be some customer issue that would need investigation that might result in a customer refund of $20-50 dollars. Setting aside that the human time required for the investigation wa…

Post by Denzel » Sat, Jan 28, 2023, 4:12 PM UTC

logicfiction wrote: post #16

Denzel wrote: post #12

Earlier quoted context omitted.

Thanks for the response. I was responding to the comment more so than advocating for adjusting your log retention. :) Looking forward to part 2. Are you able to reconcile some of the numbers and calculations in the article for me? (Understanding that you don't want to reveal any confidential info.) I see: - 31 PB data + 10 PB application logs = 41 PB logs (uncompressed json) costs 7-figures (say ~$5M) - 41 PB logs *…

The costs I talk about in the estimate in this post are for the remaining cost of each stored file. We have S3Inventory dumping metadata of all the files in specific buckets weekly, so I had written a job that calculated the exact remaining cost of each file, accounting for lifecycle events like moving to infrequent access storage in S3 and the eventual deletion of the file. So it’s sort of the “potential energy” ver…