Am I the only one that can’t reach the “save and exit” privacy button on mobile?
It’s hard for me to think that this is not intentional when the “Accept all” is usable but the alternative isn’t...
Monitoring demystified: A guide for logging, tracing, metrics
31–40 of 93 posts
Re: Monitoring demystified: A guide for logging, tracing, metrics
#32Earlier quoted context omitted.
Twice I fell into the trap of datadog, having paid more than $300 for a single month in each case. The simplicity of it, dashboards, notebooks, logs etc, is what makes it so appealing though.
If it saves you more than an hour or two, $300 seems super worth it for a system making you money...
If you're in an established company making money - yes. If you're bootstrapping a service and counting on $50 total monthly cost while initial users are signing up - no.
Re: Monitoring demystified: A guide for logging, tracing, metrics
#33A few things I have learnt along the way: Logs are great, but only once you've identified the problem. If you are searching through logs to _find_ a problem, its far too late. Processing/streaming logs to get metrics is a terrible waste of time, energy and money. Spend that producing high quality metrics directly from the apps you are looking after/writing/decomming (example: dont use access logs to collect 4xx/5xx a…
> Processing/streaming logs to get metrics is a terrible waste of time, energy and money. Spend that producing high quality metrics directly from the apps you are looking after/writing/decomming
Yeah nah, but, okay, nah yeah.
Generating metrics in the app is much more intrusive, and requires that you figure out the metrics you need ahead of time. It adds dependencies, sockets, and threads to your app.
Unless you're very careful, it's also easy to end up double-aggregating, computing medians of medians and other meaningless pseudo-statistics - if you're using the Dropwizard Metrics library, for example, you've already lost.
If you output structured log events, where everything is JSON or whatever and there are common schema elements, you can easily pull out the metrics you need, configure new ones on the fly, and retrospectively calculate them if you keep a window of log history.
When i've worked on systems with both pre- and post-calculated metrics, the post-calculated metrics were vastly more useful.
The huge, virtually showstopping, caveat here is that there is lots of decent, easy-to-use tooling for pre-calculated metrics, and next to nothing for post-calculated metrics. You can drop in some libraries and stand up a couple of servers and have traditional metrics going in a day, with time for a few games of table tennis. You need to build and bodge a terrifying pile of stuff to get post-calculated metrics going.
Anyway if there's a VC reading this with twenty million quid burning a hole in their pocket who isn't fussy about investing in companies with absolutely no path to profitability, let me know, and i'll do a startup to fix all this. I'll even put the metrics on the blockchain for you, guaranteed street cred.
Re: Monitoring demystified: A guide for logging, tracing, metrics
#34A lot of excellent information in that blog post and linked from it... but if you're wondering where to start: 1. Write good logs... not too noisy when everything is running well, meaningful enough to let you know the key state or branch of code when things deviate from the good path. Don't worry about structured vs unstructured too much, just ensure you include a timestamp, file, log level, func name (or line number…
From the post "The key, he says, is using the right transaction identifiers so that calls can be traced across components, services, and queues". I think this is a key feature not many people implement especially in today's world of over blown micro services, having a transaction id from the time the request hits the reverse-proxy till the database write is so helpful in debugging, saves a ton of time.
100% if you manage to get opentrace to work, it is a brilliant debug tool
Re: Monitoring demystified: A guide for logging, tracing, metrics
#35We log extensively. Here are some of my thoughts it - at least in C++, the requirement to be able to log from pretty much anywhere can lead to messy code that either passes a reference to your logger to all classes that might possibly need it, or you've got an extern global somewhere. Yuck. - logging can enable laziness. Being able to log that something weird happened can be considered a sufficient substitute for pro…
I'd disagree with 2 and 4.
2. Given a large enough system you will encounter situations where the only action you can take is to log "this really shouldn't happen" and try to roll back as cleanly as possible. This may be due to either complexity or a bug manifesting in a layer completely different than where it occurred (I've seen a null reference crash on "if(foo) foo->bar();" in the past)
4. I believe loggers should ideally know as little as possible about your logs. Logs can be rotated externally, can be buffered and sent to other hosts without touching the disk, can be ignored. Ideall, the system should care, not the app.
Re: Monitoring demystified: A guide for logging, tracing, metrics
#36A few things I have learnt along the way: Logs are great, but only once you've identified the problem. If you are searching through logs to _find_ a problem, its far too late. Processing/streaming logs to get metrics is a terrible waste of time, energy and money. Spend that producing high quality metrics directly from the apps you are looking after/writing/decomming (example: dont use access logs to collect 4xx/5xx a…
> Processing/streaming logs to get metrics is a terrible waste of time, energy and money. Spend that producing high quality metrics directly from the apps you are looking after/writing/decomming The other side is that I don't know what metrics I'll want until later. When do you think it's better to pull metrics from structured logs vs generating metrics in app?
I think the only times it ever really makes sense to use logs to generate metrics are fairly limited:
1. You haven't yet instrumented the application with metrics yet.
2. The logs are from a third party tool that don't emit metrics
3. The log format is well defined and doesn't change (I'd still prefer native metrics)
Otherwise the issue is that logging messages can and do change over the lifetime of an application. Relying on the content of the log for metrics becomes an implicit API that's not obvious to developers working on the code. I've seen issues of broken monitoring and alerting because a refactor changed log formatting and content. Much better to be explicit about metrics and instrument them directly.
Re: Monitoring demystified: A guide for logging, tracing, metrics
#37Earlier quoted context omitted.
Twice I fell into the trap of datadog, having paid more than $300 for a single month in each case. The simplicity of it, dashboards, notebooks, logs etc, is what makes it so appealing though.
If it saves you more than an hour or two, $300 seems super worth it for a system making you money...
It depends where you are in the world. When I was working in Switzerland, most SaaS pricing were no-brainers for us. But since I work in Latin America for small companies with local costumers, all the different services and tools you might want to use, with prices targeted at "western" customers, much more quickly add up to the equivalent of having multiple people on staff full time.
Still it is often not worth to roll your own, so it is nice to have alternatives for different price points and company scales.
Re: Monitoring demystified: A guide for logging, tracing, metrics
#38It’s weird to see the stuff by Jay Kreps (of Kafka ~fame~) listed in the logs section. His writing is specifically _not_ about logs the observability tool, but logs the data structure such as you’d see at the heart of a database.
Very true. Jay Krep's log is completely unrelated to the topic of this article. This added to my feeling that this "guide" is rather a collection of fragments put together without a real understanding of the subject from the author.
Re: Monitoring demystified: A guide for logging, tracing, metrics
#39Earlier quoted context omitted.
From the post "The key, he says, is using the right transaction identifiers so that calls can be traced across components, services, and queues". I think this is a key feature not many people implement especially in today's world of over blown micro services, having a transaction id from the time the request hits the reverse-proxy till the database write is so helpful in debugging, saves a ton of time.
100% if you manage to get opentrace to work, it is a brilliant debug tool
You can also just propagate a uuid throughout your system. I've used both uuids and opentrace. Just don't get hung up if opentrace seems too complex.
Re: Monitoring demystified: A guide for logging, tracing, metrics
#40A few things I have learnt along the way: Logs are great, but only once you've identified the problem. If you are searching through logs to _find_ a problem, its far too late. Processing/streaming logs to get metrics is a terrible waste of time, energy and money. Spend that producing high quality metrics directly from the apps you are looking after/writing/decomming (example: dont use access logs to collect 4xx/5xx a…
I miss the powerful metrics and logging systems that I used in Amazon.
> Processing/streaming logs to get metrics is a terrible waste of time, energy and money. > Spend that producing high quality metrics directly from the apps
Absolutely not. Most application metric systems generate metrics as text strings with a simple format that is parsed by the metric collector.
This is what we also call a structured log. Parsing such text strings takes very little CPU.
All logs and metrics represent events. A good approach is to prefer numerical values where possible, but only for quantities that are comparable. Metrics are for the "how many?" question.
But never forget to log text events, because you need to answer the "what happened?" question.
Don't be afraid of generating too many different metrics but avoid too frequent datapoints and unnecessary verbosity in logs.
Never dump complex objects "just in case". Treat overlogging and underlogging as a bug.
Spend time every day in reviewing the metric dashboards and improve them constantly.
If it takes more that 10 seconds do add a new non-obvious chart (e.g. to calculate a ratio between 2 metrics or a percentile or other computation) throw away your charting system.
Lying with numbers is very easy: always look at distributions, not just instant values. Some metrics must be represented as percentiles and min/avg/max are meaningless.
Percentiles are good for ignoring meaningless outliers, but always count the outliers to ensure that you are not ignoring meaningful data. Especially during incidents.
Metrics and text logs tell a story together. Process, correlate and visualize them together as much as possible.