Monitoring demystified: A guide for logging, tracing, metrics • Hacker News Board

Re: Monitoring demystified: A guide for logging, tracing, metrics

#41

Post by aloknnikhil » Fri, Jul 31, 2020, 2:17 PM UTC

kasey_junk wrote: post #29

It’s weird to see the stuff by Jay Kreps (of Kafka ~fame~) listed in the logs section. His writing is specifically _not_ about logs the observability tool, but logs the data structure such as you’d see at the heart of a database.

No. The original Kafka paper does talk about logs in the observability sense as a premise to solve the aggregation problem.

https://cs.uwaterloo.ca/~ssalihog/courses/papers/netdb11-fin...

> There is a large amount of “log” data generated at any sizable internet company. This data typically includes (1) user activity events corresponding to logins, pageviews, clicks, “likes”, sharing, comments, and search queries; (2) operational metrics such as service call stack, call latency, errors, and system metrics such as CPU, memory, network, or disk utilization on each machine. Log data has long been a component of analytics used to track user engagement, system utilization, and other metrics.

> We have built a novel messaging system for log processing called Kafka [18] that combines the benefits of traditional log aggregators and messaging systems....Kafka provides an API similar to a messaging system and allows applications to consume log events in real time.

Re: Monitoring demystified: A guide for logging, tracing, metrics

#42

Post by jrott » Fri, Jul 31, 2020, 2:19 PM UTC

dkersten wrote: post #22

buro9 wrote: post #2

A lot of excellent information in that blog post and linked from it... but if you're wondering where to start: 1. Write good logs... not too noisy when everything is running well, meaningful enough to let you know the key state or branch of code when things deviate from the good path. Don't worry about structured vs unstructured too much, just ensure you include a timestamp, file, log level, func name (or line number…

I’d add to no. 1 by saying include a correlation id or request id so you have a way to filter the logs into a single linear stream related to the same action.

Absolutely constantly being able to get a single linear stream is when tracing becomes super powerful.

Re: Monitoring demystified: A guide for logging, tracing, metrics

#43

Post by kasey_junk » Fri, Jul 31, 2020, 2:23 PM UTC

aloknnikhil wrote: post #41

kasey_junk wrote: post #29

It’s weird to see the stuff by Jay Kreps (of Kafka ~fame~) listed in the logs section. His writing is specifically _not_ about logs the observability tool, but logs the data structure such as you’d see at the heart of a database.

No. The original Kafka paper does talk about logs in the observability sense as a premise to solve the aggregation problem. https://cs.uwaterloo.ca/~ssalihog/courses/papers/netdb11-fin... > There is a large amount of “log” data generated at any sizable internet company. This data typically includes (1) user activity events corresponding to logins, pageviews, clicks, “likes”, sharing, comments, and search queries; (2)…

A quote from the LinkedIn blog post linked in the article:

“But before we get too far let me clarify something that is a bit confusing. Every programmer is familiar with another definition of logging—the unstructured error messages or trace info an application might write out to a local file using syslog or log4j. For clarity I will call this "application logging". The application log is a degenerative form of the log concept I am describing”

Re: Monitoring demystified: A guide for logging, tracing, metrics

#44

Post by dkarl » Fri, Jul 31, 2020, 2:24 PM UTC

eatonphil wrote: post #39

KaiserPro wrote: post #34

Earlier quoted context omitted.

100% if you manage to get opentrace to work, it is a brilliant debug tool

You can also just propagate a uuid throughout your system. I've used both uuids and opentrace. Just don't get hung up if opentrace seems too complex.

I agree with this wholeheartedly. You can even define a standard and let downstream services opt in over time. Simple wins like this should not be put off because "someday" we're going to implement a complex distributed tracing solution.

Re: Monitoring demystified: A guide for logging, tracing, metrics

#45

Post by a10c » Fri, Jul 31, 2020, 2:49 PM UTC

say_it_as_it_is wrote: post #4

Is there an open source solution for processing streams of structured and unstructured logs and routing then onward? I see solutions for moving logs to elastic or Kafka but nothing for evaluating the log.

https://vector.dev/ sounds pretty close

Re: Monitoring demystified: A guide for logging, tracing, metrics

#46

Post by _i4zn » Fri, Jul 31, 2020, 2:55 PM UTC

viraptor wrote: post #21

buro9 wrote: post #2

A lot of excellent information in that blog post and linked from it... but if you're wondering where to start: 1. Write good logs... not too noisy when everything is running well, meaningful enough to let you know the key state or branch of code when things deviate from the good path. Don't worry about structured vs unstructured too much, just ensure you include a timestamp, file, log level, func name (or line number…

> Grafana Cloud offers a hosted Grafana, Prometheus metrics scraping and storage, and Log tailing and storage (via Loki) I haven't looked at their pricing before, but for small-ish environments, their standard plan looks really good and simple. None of the "per host, but also per function, and extra for each feature, and extra for usage" approach like other providers (datadog, I'm looking at you).

> None of the "per host, but also per function, and extra for each feature, and extra for usage" approach like other providers (datadog, I'm looking at you).

I was thinking "God, this is exactly why I hate Datadog" as I was reading your description and got a great laugh when I reached the end. Their billing is absolutely byzantine.

I don't know that I've ever seen a company that had such a stark difference between great engineering/product and awful business/sales practices. Their product is really the best turn-key option out there, but I'm always hesitant to use its features without double checking it's not going to add 50% to my bill. Their sales teams are some of the worst I've dealt with, and I deal with a lot of vendors. They're starting to get a really bad reputation as well.

Re: Monitoring demystified: A guide for logging, tracing, metrics

#47

Post by yellow_mixer » Fri, Jul 31, 2020, 3:00 PM UTC

ysoft wrote: post #24

buro9 wrote: post #2

A lot of excellent information in that blog post and linked from it... but if you're wondering where to start: 1. Write good logs... not too noisy when everything is running well, meaningful enough to let you know the key state or branch of code when things deviate from the good path. Don't worry about structured vs unstructured too much, just ensure you include a timestamp, file, log level, func name (or line number…

I don't think loki is production ready. Needs more work. It's going to be great with grafana though

What else do you think it needs. I’ve been playing around with v1.5 the past few days in Grafana and I like it’s simplicity. Grafana now has a Loki datasource which is nice.

Re: Monitoring demystified: A guide for logging, tracing, metrics

#48

Post by dillonmckay » Fri, Jul 31, 2020, 3:07 PM UTC

dkarl wrote: post #44

eatonphil wrote: post #39

Earlier quoted context omitted.

You can also just propagate a uuid throughout your system. I've used both uuids and opentrace. Just don't get hung up if opentrace seems too complex.

I agree with this wholeheartedly. You can even define a standard and let downstream services opt in over time. Simple wins like this should not be put off because "someday" we're going to implement a complex distributed tracing solution.

So, in terms of full implementation, is the constraint the dev time to implement, or getting various factions to agree to something?

I guess, is it political, or technical?

Asking for a friend.

Thanks!

Re: Monitoring demystified: A guide for logging, tracing, metrics

#49

Post by bostonsre » Fri, Jul 31, 2020, 3:18 PM UTC

_i4zn wrote: post #46

viraptor wrote: post #21

Earlier quoted context omitted.

> Grafana Cloud offers a hosted Grafana, Prometheus metrics scraping and storage, and Log tailing and storage (via Loki) I haven't looked at their pricing before, but for small-ish environments, their standard plan looks really good and simple. None of the "per host, but also per function, and extra for each feature, and extra for usage" approach like other providers (datadog, I'm looking at you).

> None of the "per host, but also per function, and extra for each feature, and extra for usage" approach like other providers (datadog, I'm looking at you). I was thinking "God, this is exactly why I hate Datadog" as I was reading your description and got a great laugh when I reached the end. Their billing is absolutely byzantine. I don't know that I've ever seen a company that had such a stark difference between gr…

I'm a customer that uses most of their tools (no network performance monitoring since it's less useful than a service mesh and no logging because we need longer history than most and cost would be prohibitive).

Is it really that expensive when compared to other vendors? Thought their newer logging tool was a lot cheaper than splunk and their apm tool for distributed tracing is also pretty cheap when compared with something like new relic. Sure it's more expensive than free tools that you need to setup yourself. But the velocity it lets your teams have is so much better than having to use something like grafana with tools like Prometheus. Again, sure it can be done for cheap, but the time it takes to manage those tools and the velocity that you lose when doing that doesn't seem like it's worth it for smaller companies but I can see it making more sense as you scale a company.

Re: Monitoring demystified: A guide for logging, tracing, metrics

#50

Post by gregwebs » Fri, Jul 31, 2020, 3:24 PM UTC

say_it_as_it_is wrote: post #4

Is there an open source solution for processing streams of structured and unstructured logs and routing then onward? I see solutions for moving logs to elastic or Kafka but nothing for evaluating the log.

You can actually do this log manipulation in fluent-bit (you can write Lua if you need to) although the forwarding cannot be routed to different locations.