Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Saving millions on logging: Finding relevant savings

product.hubspot.com

Post reply on HN
Print view View original

11–20 of 24 posts

Re: Saving millions on logging: Finding relevant savings

#11

Post by orf »

logicfiction wrote: post #7
orf wrote: post #6

Intercepting network traffic like this is an interesting approach to the problem. If each service has a unique IAM role, which it definitely should do, wouldn’t you be able to track this via a combination of cloudtrail and proper resource tags?

(Author here). Yes I believe you are correct with regards to tracking application utilization of say EC2 and other AWS resources. The post fails to mention this system is also tracking internal data dimensions like customer ids, such that we can also use this sampled data to estimate the cost of customers (and joining that with tiers of customers, and so forth). I'm also not sure if that would allow us to attribute t…

When it comes to shared resources like a database cluster, you’re making the assumption that usage is correlated with number of connections.

Is this always true? Typically the shared resources you care about are CPU, memory and disk. I would say an application issuing fewer, much heavier queries is using the shared resource more than an application that issues more really simple queries. And this doesn’t correlate much to disk usage right?

There isn’t really a good solution to this. You can use a combination of query sampling and per-app databases to correlate this better.

Great post though, this is something we’ve been dealing and experimenting with recently.

Re: Saving millions on logging: Finding relevant savings

#12

Post by Denzel »

logicfiction wrote: post #9
Denzel wrote: post #3

Earlier quoted context omitted.

Yes, shipping computations instead of data is a reasonable design goal. Your proposed system only works when the predicate is independent across all logs though, correct? If you have to correlate or join your logs to anything, then this model becomes more complex. Not to mention, you're adding an additional performance tax to your prod machines which could be more costly than shipping logs to a centralized store. (A…

That's a fair criticism in the edit. Part 2 will cover that a bit more. I did run analysis on the types of queries users ran against the data and what parts of the timeseries were used, which informed a bit of our solution. I don't want to give away too much, but lifecycle retention adjustment ends up being relatively lower value (but still worthwhile) compared to general space savings.

Thanks for the response. I was responding to the comment more so than advocating for adjusting your log retention. :) Looking forward to part 2.

Are you able to reconcile some of the numbers and calculations in the article for me? (Understanding that you don't want to reveal any confidential info.) I see:

- 31 PB data + 10 PB application logs = 41 PB logs (uncompressed json) costs 7-figures (say ~$5M)

- 41 PB logs * 5% ORC compression = ~ 2 PB logs (compressed ORC) costs low 6-figures (say ~$300k)

I don't know what timeframe that cost is measured over. But that brings us to $300k / 2 PB = $0.15 / GB which is far above S3's quoted costs so I must be missing something.

Re: Saving millions on logging: Finding relevant savings

#13

Post by dilyevsky »

jeffbee wrote: post #2

It feels like logging is misunderstood. Critical revenue or audit logs need to be centralized, but debug logs don’t. Logging debug logs to local storage and deleting it after nobody looks at it-the lifecycle of at least 99.999% of informational log statements-costs almost nothing. Another benefit is that pushing your predicate out to your edge nodes works far better than trying to get acceptable performance from cent…

Because you can’t sell that as a saas product

Re: Saving millions on logging: Finding relevant savings

#14

Post by olliej »

I am lost at how you can have 20% of your storage costs be for logging, and not immediately say that at minimum you are persisting too many logs, and probably logging too much in the first place.

I get that modern tech companies log every movement and interaction a user has with an app, far beyond any amount that is reasonable, but surely at some point you can go “we probably don’t need this”.

It shouldn’t be a matter of “let’s compress the logs”, it should be a “are we even using these logs”.

Re: Saving millions on logging: Finding relevant savings

#15

Post by SamuelAdams »

olliej wrote: post #14

I am lost at how you can have 20% of your storage costs be for logging, and not immediately say that at minimum you are persisting too many logs, and probably logging too much in the first place. I get that modern tech companies log every movement and interaction a user has with an app, far beyond any amount that is reasonable, but surely at some point you can go “we probably don’t need this”. It shouldn’t be a matte…

I agree with you, but never underestimate fear of litigation. Legal and HR departments everywhere love to have extremely detailed logs for years that can refute ex-employees claims about one thing or another.

Re: Saving millions on logging: Finding relevant savings

#16

Post by logicfiction »

Denzel wrote: post #12
logicfiction wrote: post #9

Earlier quoted context omitted.

That's a fair criticism in the edit. Part 2 will cover that a bit more. I did run analysis on the types of queries users ran against the data and what parts of the timeseries were used, which informed a bit of our solution. I don't want to give away too much, but lifecycle retention adjustment ends up being relatively lower value (but still worthwhile) compared to general space savings.

Thanks for the response. I was responding to the comment more so than advocating for adjusting your log retention. :) Looking forward to part 2. Are you able to reconcile some of the numbers and calculations in the article for me? (Understanding that you don't want to reveal any confidential info.) I see: - 31 PB data + 10 PB application logs = 41 PB logs (uncompressed json) costs 7-figures (say ~$5M) - 41 PB logs *…

The costs I talk about in the estimate in this post are for the remaining cost of each stored file. We have S3Inventory dumping metadata of all the files in specific buckets weekly, so I had written a job that calculated the exact remaining cost of each file, accounting for lifecycle events like moving to infrequent access storage in S3 and the eventual deletion of the file. So it’s sort of the “potential energy” version of the cost of our stores files. If we take no action they will aggregate to a certain amount of money.

I reckon you may be looking at the monthly cost of storage per gigabyte which is why the number doesn’t seem to make sense. Our retention policy started off at about 2 years, so the remaining lifetime per file amortizes out to much more than 1 month.

Also worth considering that we have a custom AWS contract, so none of our actual numbers are the publicly advertised rates and probably won’t entirely math out if you try to ballpark with those numbers.

Re: Saving millions on logging: Finding relevant savings

#17

Post by logicfiction »

olliej wrote: post #14

I am lost at how you can have 20% of your storage costs be for logging, and not immediately say that at minimum you are persisting too many logs, and probably logging too much in the first place. I get that modern tech companies log every movement and interaction a user has with an app, far beyond any amount that is reasonable, but surely at some point you can go “we probably don’t need this”. It shouldn’t be a matte…

You’re right, it’s not an either or for this as we tackle both less total data and making it smaller, although I probably failed to clarify that in this first post.

At 20% of storage costs, the should makes a lot of sense to focus on. Once it becomes 1% of storage costs it’s maybe not as problematic though. The magnitude to which, “let’s compress the logs” changes how much something like “am I logging too much” matters is important. Taking it to the absurd, if logging storage were free why not retain all logs. And if logging is cheap, why invest in complicated guardrails for what qualifies as important logs.

A specific consideration for us is organizational inertia. We have a lot of teams using infrastructure in a lot of ways both intended and unintended. One thing, for better or worse, that has been emphasized for us is developer velocity. Which includes things like abstracting “do I need to log this” from most engineers. We have some guardrails to alert if you do some egregious volume.

I think we do often opt for non-invasive infra solutions first because they have much shorter delivery times and less risk of stalling on long-tail outliers. They avoid very expensive organizational costs of buy in and team-level migration. I’m not suggesting this is the best organizational model, but it also transcends one team’s influence.

That ends up circling back around to the start of the problem. If we can transparently reduce the cost burden of some heavily-used internal infrastructure within the same relative magnitude of applying a paradigm shift on the usage of the same internal infrastructure, the former wins out.

Re: Saving millions on logging: Finding relevant savings

#18

Post by logicfiction »

orf wrote: post #11
logicfiction wrote: post #7

Earlier quoted context omitted.

(Author here). Yes I believe you are correct with regards to tracking application utilization of say EC2 and other AWS resources. The post fails to mention this system is also tracking internal data dimensions like customer ids, such that we can also use this sampled data to estimate the cost of customers (and joining that with tiers of customers, and so forth). I'm also not sure if that would allow us to attribute t…

When it comes to shared resources like a database cluster, you’re making the assumption that usage is correlated with number of connections. Is this always true? Typically the shared resources you care about are CPU, memory and disk. I would say an application issuing fewer, much heavier queries is using the shared resource more than an application that issues more really simple queries. And this doesn’t correlate mu…

Your observations are correct. I wouldn’t portray it as an ideal system, just best effort. In the end we care more about the finer details being good funnels to follow up on rather than being exact. We know our real costs of resources, which is important for finance and budget. And then we have the approximate attributions from the sampling which narrow things down enough to focus diagnosis when needed.

I would have to read more into how it intercepts some of our database calls to confirm if it tries to weight for execution time where it wires into database client code, which is probably useful and could help to a degree to approximate utilization.

I think in practice it’s a bit uncommon for the heaviest user to also be a sparse user in terms of volume. But I can also admit there are quirks to how it samples and I once personally spent a couple days tracking down a surprising cost of an application I owned only to later confirm it was a data flaw in how we were doing this sampled attribution (in this case the heaviest users were un-instrumented infra processes that can’t just wire in our java cost attribution library, making it artificially look like my app was the heaviest user).

Re: Saving millions on logging: Finding relevant savings

#19

Post by logicfiction »

nrivoli wrote: post #10

I would like to know more about the kubernetes databases, what kind, challenges, how are the fault domains configured and etc. Also is not clear to me how intercepting calls helped you to figure out the offending services?

I would have to defer to one of my colleagues for most of the details on running data infrastructure on Kubernetes, I’m not that close to that domain. The major ones we run are HBase, Vitess (MySql), Kafka, Elasticsearch, Memcached, and Zookeeper.

As for the call sampling/interception, that did not factor into discovering the high cost buckets in the logging case study. It was mostly relevant to generally describing how we track costs and it ends up being useful in other scenarios. For example it could be used to assess the estimated unit economics of customers subscribed into a specific product tier.

We also have the death star microservice model, so even relatively simple attribution can be helpful when you want to run a query like “for my team which owns 30 applications, tell me the monthly attributed cost grouped by resource” and that will be able to return all the associated database and cloud costs.

Re: Saving millions on logging: Finding relevant savings

#20

Post by hinkley »

olliej wrote: post #14

I am lost at how you can have 20% of your storage costs be for logging, and not immediately say that at minimum you are persisting too many logs, and probably logging too much in the first place. I get that modern tech companies log every movement and interaction a user has with an app, far beyond any amount that is reasonable, but surely at some point you can go “we probably don’t need this”. It shouldn’t be a matte…

Using logs to derive telemetry is a short term strategy, not a long term one. This degree of logging suggests they’ve been doing everything with logs.
Post reply on HN