101–109 of 109 posts
Earlier quoted context omitted.
My systems scale to somewhere around 200,000+ machines at peak and seem to do fine.
How many customers is that serving, 20 per machine? (Or whatever a customer might be in that case)
Earlier quoted context omitted.
This is a reasonable first pass answer, but there's more nuance to this... > What it does not do is separate this information Logging at scale should really be structured, which means that you can trivially differentiate between different types of log message. You also get more dimensions all represented in that structure. > limits the type of metrics which can be reported to be those expressible in a message text fi…
I think what you're saying is that you can make a logging system LARP metrics. At the end it's logging on fd 1 and 2 and metrics are usually over http, but ofc you can dump "metrics" into stout, it's not as practical with what tools are built for what. In my local fun projects that run on my machine I might dump metrics into the logs because it's practical, but it doesn't make it "right".
Also it's not about logs role playing as metrics, I'm saying you can literally turn one into the other, in both cases, and there are valid use cases for that.
Checkpointing the model every N iterations/epochs/batches has a similar problem - you may end up saving very few checkpoints and risk losing work or waste a lot of time/space with lots of checkpoints.
So I've often found myself implementing some kind of monitoring and checkpointing callbacks based on time, e.g., reporting every half an hour, checkpointing every two hours, etc.
This post falls into a common trap; conflating logging with metrics. Log interesting things, where interesting is defined as context outside what the "happy path" execution performs. Collect and make available system metrics, such as invocation counts, processing time histograms, etc., to make available what the post uses log statements to disseminate same.
Thanks for taking the time to reply! I'm relatively new to working on this type of system (large scale, event driven) and half posted because I know there are people on HN way better than me at this, and was curious about their opinions. In the end, what's the difference between a log and a metric? Is one structured, and one unstructured? Is one a giant blob of text, and the other stored in a time series db? At the m…
The goals. The goals of the activity is the difference.
The goal of logging is diagnostics and trouble-shooting (when did this break, how often do we see this type of failure, etc).
The goal of metrics is to aid in capacity planning (are we close to running out of RAM, do we exceed 80% CPU too often, etc).
> You and the other commenters have given me the vocabulary to dig more into this area on the internet though.
Read this first; it is a short read (taxonomy of logging, basically): https://www.lelanthran.com/chap10/content.html
Earlier quoted context omitted.
The breakdown used by OTel isn’t all that bad: https://opentelemetry.io/docs/concepts/signals/ In essence: Logs mark some event in the system. Metrics model some measurable, quantifiable state. In high volume systems both can then be observed through various sampling techniques. A key item is that sampling is good to handle separately to application logic creating those signals as it may change over time or be dynami…
I think the common confusion boils down to: > The moment of capturing a measurement is known as a metric event Which suspiciously reads like a log. In practice, a metric is an aggregate of events (the "metric events") when you're not interested in the individual event but, but in the aggregate itself. For practical reasons this is not implemented with logs but with more primitive technical events emission. This is no…
> The moment of capturing a measurement is known as a metric sample.
The mental model I hold is the metric is the actual value. This may be discrete (e.g. a packet counter) or some continuous value (e.g. a voltage in you ECG example). It can then be observed at some time/value delta interval or summarised into other time series based on what you're hoping to capture.
Earlier quoted context omitted.
I've had colleagues try this. It rarely works. Logging every if end up introducing a huge amount of overhead, both in terms of processing power, but especially in terms of storage. You almost always end up having to filter based on some sort of log level that you then turn off by default in production. The problem with that is that you're now required to reproduce the issue after turning on the logging, and if you al…
Something I've done in the past is send some logs to BigQuery for cheap mass storage and others to Grafana for fast querying and use in live dashboards. Basically a filter rule in our logging agent to send different events to different destinations. I think with some more hacking I could get both datasources into the same Grafana frontend...
The log filter is all of a sudden part of the application, but managed somewhere else. Everybody is now scared of touching the log lines because who knows what filters have been configured. You suddenly have to debug your log setup, and who logs the decisions about the logs that were filtered?
We already have a place to put that logic. It's the application.
One way to reframe this is: "as a user [of the logs], what might I want to know?" In my experience, this post is often right (and the logs are often wrong). There's a tendency to either log too much or log too little - if only a few items are getting processed, it's fine and maybe even good to log all 7 of them. But if many, many are getting processed - you'll experience semantic overload as a reader of the logs. Wha…
The "compressed form" of logs you're describing here is really just metrics...
Earlier quoted context omitted.
> I wonder how do they log mission critical things in general. For instance, how often does a flight data recorder (FDR) log every state of mechanical components? Surely, they can't wait until something "interesting" to happen, right? There are different types of logging. What you describe could be defined as an audit log intrinsic to system operation, which is quite a different thing than what the article describes.
Oh, I see. My bad then. Could you expand a bit more?
> Oh, I see. My bad then. Could you expand a bit more?
Sure, I'll do my best.
The terms "logging" and "logs" are overloaded in the software industry and often used assuming context is known. Below are four examples illustrating same.
# Program Logs
This type of logging is what the article discusses and is what many developers are most familiar with. Usually, each entry is associated with a level (such as "debug", "info", "error", etc.) and capture state relevant to program execution at the point of log emission during the call tree is evaluation.
Usually they are used for postmortem analysis when a problem is discovered.
# Audit Logs
This type of logging serves to record changes in the persistent representation of key abstractions, often due to regulatory requirements. Your example of "a flight data recorder" is a great exemplar.
Audit logs for an entity are often independent of other entities and may not support the ability to replay the changes.
# Write-Ahead Logs[0]
This type of logging is frequently used as an implementation detail for various database technologies, such as RDBMS's.
WAL's are most often employed to address database crash recovery and intimately involved with transaction management.
# Event Sourcing[1]
While Event Sourcing might not be immediately thought of as a form of logging, it becomes clear it is when considering how the events are stored. Most discussions of Event Sourcing either describe it in terms of an Audit Log (like the referenced article does) or as an "append-only log."
Earlier quoted context omitted.
The "compressed form" of logs you're describing here is really just metrics...
it's not, though. sometimes you need actual active logs to tell you that something is progressing.