Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fast and flexible observability with canonical log lines

stripe.com

Post reply on HN
Print view View original

31–40 of 41 posts

Re: Fast and flexible observability with canonical log lines

#31

Post by traskjd »

msoad wrote: post #3

is there any legal restriction of how long you can keep internal systems logs? if it's done right they don't contain PIIs but they _can_ be used to track people if you have enough logs.

There's some vague stuff. For example, GDPR requires you only keep data for a "reasonable" period of time. So many many years would likely not be reasonable in most logging scenarios.

I suspect as a payment processor though, being able to look back far when investigating breeches etc would be important.

Re: Fast and flexible observability with canonical log lines

#34

Post by spearo77 »

brandur wrote: post #28
manigandham wrote: post #22

Strange that they went with plain text when the industry is converging on (newline delimited) JSON logs for structured data. This also serves as the backbone of observability with metrics and tracing also being folded into and output as JSON. Call them events and you can claim all the event-sourcing buzzwords too.

I wouldn't put too emphasis on the plain text — we started logging back when carrying everything via JSON would've been going against the grain. These days it might've gone the other way (I'm not sure). One point that I'd try to convey is that the canonical line technique works for any kind of structured format. We use logfmt in all our examples, but JSON would work just as well.

I searched for logfmt and found your website – https://brandur.org/logfmt

What I can't find described for logfmt is how to handle escaping special characters in the keys or values. Specifically the quotes and equals.

JSON defines this very well, but Splunk simple key=value logging formats suffer from escaping and parsing problems.

Re: Fast and flexible observability with canonical log lines

#35

Post by mirekrusin »

spearo77 wrote: post #34
brandur wrote: post #28

Earlier quoted context omitted.

I wouldn't put too emphasis on the plain text — we started logging back when carrying everything via JSON would've been going against the grain. These days it might've gone the other way (I'm not sure). One point that I'd try to convey is that the canonical line technique works for any kind of structured format. We use logfmt in all our examples, but JSON would work just as well.

I searched for logfmt and found your website – https://brandur.org/logfmt What I can't find described for logfmt is how to handle escaping special characters in the keys or values. Specifically the quotes and equals. JSON defines this very well, but Splunk simple key=value logging formats suffer from escaping and parsing problems.

You could use yaml serialiser, which leaves non special strings as is and quotes special ones. But this approach can break naive grepping.

Re: Fast and flexible observability with canonical log lines

#37

Post by jsmeaton »

ianstormtaylor wrote: post #16

Great article! I always love hearing Stripe talking about their internals. I've been using this practice and I agree that it's incredibly useful. I think because people tend to think in terms of "logs", they end up overlooking the much more useful construct of "canonical logs". Many fine-grained logs themselves are almost always less useful than the fewer fully-described canonical logs. Other observability tools ofte…

Thanks for talking about Honeycomb, I’ve been looking out for peoples experiences as it looks like quite a nice tool.

We’ve just moved over to structured event based logging, but we’re still using tools like Sumo, newrelic, etc and it’s not all nicely tied together.

I’ve been looking at Honeycomb to either replace or augment our existing tools but had little to go on besides the marketing pages.

Re: Fast and flexible observability with canonical log lines

#38

Post by lizthegrey »

jsmeaton wrote: post #37
ianstormtaylor wrote: post #16

Great article! I always love hearing Stripe talking about their internals. I've been using this practice and I agree that it's incredibly useful. I think because people tend to think in terms of "logs", they end up overlooking the much more useful construct of "canonical logs". Many fine-grained logs themselves are almost always less useful than the fewer fully-described canonical logs. Other observability tools ofte…

Thanks for talking about Honeycomb, I’ve been looking out for peoples experiences as it looks like quite a nice tool. We’ve just moved over to structured event based logging, but we’re still using tools like Sumo, newrelic, etc and it’s not all nicely tied together. I’ve been looking at Honeycomb to either replace or augment our existing tools but had little to go on besides the marketing pages.

Honeybee here. Feel free to just try it, there's a 14 day free trial, and a free community edition for small amounts of data :) experiment away, and our community slack is super friendly!

Re: Fast and flexible observability with canonical log lines

#39

Post by lizthegrey »

RhodesianHunter wrote: post #26
ianstormtaylor wrote: post #16

Great article! I always love hearing Stripe talking about their internals. I've been using this practice and I agree that it's incredibly useful. I think because people tend to think in terms of "logs", they end up overlooking the much more useful construct of "canonical logs". Many fine-grained logs themselves are almost always less useful than the fewer fully-described canonical logs. Other observability tools ofte…

How are these "events" different from sending this data to a metrics aggregator?

Because the events are correlated with each other. When you send individual metrics, the fact that you incremented A and B together isn't visible.

Re: Fast and flexible observability with canonical log lines

#40

Post by spimmy »

twic wrote: post #20
ianstormtaylor wrote: post #16

Great article! I always love hearing Stripe talking about their internals. I've been using this practice and I agree that it's incredibly useful. I think because people tend to think in terms of "logs", they end up overlooking the much more useful construct of "canonical logs". Many fine-grained logs themselves are almost always less useful than the fewer fully-described canonical logs. Other observability tools ofte…

Honeycomb's CEO on these "wide events": > The most effective way to structure your instrumentation, so you get the maximum bang for your buck, is to emit a single arbitrarily wide event per request per service hop. > We're talking wiiiide. We usually see 200-500 dimensions in a mature app. But just one write. https://twitter.com/mipsytipsy/status/1042817542648082432

^ CTO now, but thanks :D
Post reply on HN