Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Logging practices I follow

16elt.com

Post reply on HN
Print view View original

71–80 of 81 posts

Re: Logging practices I follow

#71

Post by darkwater »

marginalia_nu wrote: post #24
darkwater wrote: post #17

Earlier quoted context omitted.

If your infra is not on-prem, yes it will cost you more money as you are generating more and more and bigger logs.

You actually have to log a damn lot to actually fill up even a single 16 Tb drive with gzip-compressed logs which typically have something like 50x compression for log data. On top of that, mechanical hard drives are pretty cheap these days. Like it's a dozen dollars per terabyte, if not less. I don't know, you're either producing just absurd amounts of logs, on the order of a hundred gigabytes a day plain text, at w…

> You actually have to log a damn lot to actually fill up even a single 16 Tb drive with gzip-compressed logs which typically have something like 50x compression for log data.

If you cannot search quickly in the logs, at least the "hot" ones (i.e. most recent) they don't make too much sense. Well, they still make sense but for other reasons, but you lose many interesting feature of logs. At $DAYJOB we *surely* needs to trim and shave a lot the logs apps are sending to the centralized ELK - which is one of the points of TFA - but we cannot just gzip the text files and be done, we need to be able to search for patterns anbd data in the logs to understand what the app is doing in certain cases (besides having metrics).

P.S. We also store them as gzipped files in an S3 bucket using warm/cold tiers, and it is certainly cheaper than using even magnetic disks.

Re: Logging practices I follow

#72

Post by dengolius »

adra wrote: post #48
masklinn wrote: post #45

Earlier quoted context omitted.

Is there a structured form of tracing? Because I feel like this contextual information should easily be part of a trace.

Logging is kinda a mess in general for contextualization. Most that support it use kV tupled appended to the log line itself. OpenTelemetry is probably the best hope of supporting a world with contextual logs, metrics, traces which imho is a good thing. OTEL logging does some opinionated things with message construction though, so caveat emptor.

From my experience, traces are not need for every one. AFAIK opentelemetry "standard" (sdk) changes very often, especially painful because of the constant critical changes in the library, a big size of compiled executable binary on the otuput...

Re: Logging practices I follow

#73

Post by marginalia_nu »

adra wrote: post #59
marginalia_nu wrote: post #6

Earlier quoted context omitted.

You have to output a lot of logs before you fill up even a single large consumer-grade hard drive, especially given logs are typically compressed when rotated. It's usually only when you involve ELK or something like that your logs start to get big. Which in turn is typically necessitated by over-complicated distributed software design. If you're at the scale where this actually matters and you're serving millions or…

Except you know when you.actually want to do something valuable with all those logs. You _should_ be creating logs (signals) to be valuable in some way (diagnostics, alerting, canaries statistics), etc. If you're just dumping logs into opaque blobs that are never looked at them sure write them to blobs to your heart's content and have fun hunting and pecking for reasons you're users are already screaming at you. That…

Depends entirely on what and why you are logging.

Is it audit logs for security or due to some regulatory requirement? Then huge blobs are fine. Desirable, even.

Transaction logs for machine-loading so you're able to replay an application's state at any given moment in time? Yeah probably gonna end up with huge blobs again.

Re: Logging practices I follow

#74

Post by giaour »

partdavid wrote: post #49
giaour wrote: post #40

Earlier quoted context omitted.

When people complain about the cost of excessive logging, they are almost certainly not thinking in terms of how much a drive costs. Services like CloudWatch are an excellent way to burn through money, though it's usually the time series storage and ingestion costs that balloon out of control.

Well, also the kind of people who worry about this are not thinking in terms of "a terabyte", like GP. It's always easy to give advice when your experience has been at a toy level.

And if you're in a position where you have to manage petabytes+ of logs using on-prem hardware, SSDs are probably a small part of your overall budget!

Re: Logging practices I follow

#75

Post by vinceguidry »

marmada wrote: post #22
vinceguidry wrote: post #4

I fight log infra all the time. I can't win the fight against structured logging anymore, so I'm now fighting against type systems and allergies to global state to make log output available everywhere. If you're going to ram structured logging down devs' throats, then the least you can do is to make it easy enough to use. I don't want to have to pass a logging object everywhere. There's like two pieces of information…

What is the reason structured logging is bad? I'm curious, as I felt like it made my life a lot easier.

Simply put, it makes logs unreadable for humans without tooling. Things primarily touched by humans should be as friendly as possible to them. Prior to the advent of structured logging, every log message had a unique visual 'fingerprint' that the eyes and brain could grok at a glance, and spot anomalies really quickly. With structured logs, everything looks the same and so you can't use the brain's inbuilt abilities to process them.

And if you did want tooling to help, then typically the formats were regular and so you could use ordinary text processing tools to help. sed, awk, grep. With structured logging putting everything into nested balanced expressions, you need parsing. Parsing works until you run across something the parser can't figure out. Say you have some Go code with a JSON logger object you're passing around. What if you want to log something but your logging object isn't passed to that function. (you could pass it everywhere but that increases the arity of every single method by 1, and are also reinventing global state poorly) You're SOL, now you're stuck with fmt.Println() and you just broke jq. No, jq does not handle this failure mode. No, Golang does not let you just spit out arbitrary JSON. Thy must use the logging object.

The only thing it helps is ingestion into databases for heavy machine processing. Which is fine, but don't make it the only or even default way software tools spit out logs. In every other way, introducing parsing into your workflow just slows it all down. The only way I can see structured logging making anyone's job easier is if they never understood how it all worked before.

Re: Logging practices I follow

#76

Post by dns_snek »

partdavid wrote: post #49
giaour wrote: post #40

Earlier quoted context omitted.

When people complain about the cost of excessive logging, they are almost certainly not thinking in terms of how much a drive costs. Services like CloudWatch are an excellent way to burn through money, though it's usually the time series storage and ingestion costs that balloon out of control.

Well, also the kind of people who worry about this are not thinking in terms of "a terabyte", like GP. It's always easy to give advice when your experience has been at a toy level.

That's unnecessarily dismissive. Handling many (or dozens, hundreds) TB worth of logs is anything but "toy level", that's more than the vast majority of businesses will generate in a decade, maybe even their lifetime.

Re: Logging practices I follow

#77

Post by partdavid »

dns_snek wrote: post #76
partdavid wrote: post #49

Earlier quoted context omitted.

Well, also the kind of people who worry about this are not thinking in terms of "a terabyte", like GP. It's always easy to give advice when your experience has been at a toy level.

That's unnecessarily dismissive. Handling many (or dozens, hundreds) TB worth of logs is anything but "toy level", that's more than the vast majority of businesses will generate in a decade, maybe even their lifetime.

And marginalia_nu, the GP I was referring to, was unnecessarily strident, concluding that others must be naïve or incompetent if they had to handle logs with "ELK or something like that" and that therefore one must have an "over-complicated distributed software design."

Don't move the goalposts to hundreds of TB--this user is giving advice to everyone based on a perspective that you're doing something wrong if all of your logs don't fit on a single hard drive; that you should "log less" if you have the "absurd" quantity of "hundreds of gigabytes" a day of logs, and who seems to think individual hard drive costs is an important driver of the cost of managing logs. Their words, not mine.

There's nothing interesting to be gained from hot takes based on naïve conceptions and lack of experience. Pointing out that giving overly-general advice based on your inexperienced best guesses and the NewEgg price list is not very useful is not "unnecessarily dismissive."

Re: Logging practices I follow

#78

Post by partdavid »

giaour wrote: post #74
partdavid wrote: post #49

Earlier quoted context omitted.

Well, also the kind of people who worry about this are not thinking in terms of "a terabyte", like GP. It's always easy to give advice when your experience has been at a toy level.

And if you're in a position where you have to manage petabytes+ of logs using on-prem hardware, SSDs are probably a small part of your overall budget!

Also true, though it's not zero. The different cost drivers/cost model between using SaaS and on-prem infrastructure for logs are interesting and drive different decisions. I have done both in large and small environments and I kind of like the SaaS model because it is easier to put cost incentives on product owners and development teams, which is who should own the P&L. On other words, if you pay $1/GB or whatever, you can get back money by logging fewer GBs. It naturally discourages "log whatever into a giant undifferentiated bucket 'in case you need it'".

You can pay less for equivalent on-prem infrastructure but it drives costs quite differently. For example, it tends to be hard to refresh that infrastructure because it doesn't make you money, so it gets worse over time. The unit cost of storage is very low, often because you make availability/durability tradeoffs that aren't even available to you from the SaaS provider or cloud service. But you will find that the Opex associated with it can be either quite high or poor, and this is hard to reflect in terms of investment by P&L owners.

You can do either approach well or poorly. The way SaaS sucks when doing it poorly is mostly that you are paying a huge amount of money. The way on-prem sucks when doing it poorly is much more complicated and is reflected by toil and tech debt across an organization (which is money but harder to tie to what would fix it), poor visibility, lack of insights, and possibly spending too much in Opex or licenses, depending on the technology. The cost of having a bunch of people do on-prem logging "right" is hard to justify, even for these "large organizations" where I guess, people think, money is free. And even if you've correctly identified and wish to fund the cost of delivering the infrastructure (which as you point out has hardware as only part of its cost), it's not like you can necessarily find the five quality engineers to run the thing. And if you could--do you really want these FTEs working on logging infrastructure or do you want them delivering revenue features?

Re: Logging practices I follow

#79

Post by valyala »

ilyt wrote: post #58
marginalia_nu wrote: post #24

Earlier quoted context omitted.

You actually have to log a damn lot to actually fill up even a single 16 Tb drive with gzip-compressed logs which typically have something like 50x compression for log data. On top of that, mechanical hard drives are pretty cheap these days. Like it's a dozen dollars per terabyte, if not less. I don't know, you're either producing just absurd amounts of logs, on the order of a hundred gigabytes a day plain text, at w…

> You actually have to log a damn lot to actually fill up even a single 16 Tb drive with gzip-compressed logs which typically have something like 50x compression for log data. Now count that for queryable data source so running a database of some sort (Elasticsearch probably for logs) 24/7 at fast enough speeds that it is ops-useful Metrics are significantly cheaper tho, at least if you use some dedicated TSDB with g…

VictoriaMetrics author here. I'm working on VictoriaLogs right now, e.g. the logging system on top of VictoriaMetrics architecture ideas. Preliminary results are promising:

- It will need much lower amounts of disk space, disk IO, CPU and RAM comparing to ElasticSearch during data ingestion.

- It will provide fast logs' querying and tailing via easy-to-use query language (LogsQL), with the ability to calculate advanced stats over the selected logs.

- It will accept data in ElasticSearch format, so existing Filebeat and Logstash setups can be switched from ElasticSearch to VictoriaLogs in a few seconds.

Re: Logging practices I follow

#80

Post by vinceguidry »

partdavid wrote: post #50
antonyt wrote: post #36

Earlier quoted context omitted.

I have the same question! I understand parent's criticisms of context objections and logging boilerplate, but I'm not following the "fight against structured logging." What are the alternatives? No logging? Unstructured logging? Why would either of those be better than structured logs?

While I personally favor no logging, almost everyone who criticizes structured logging would prefer unstructured logging so they can make it someone else's job to restructure it (i.e. index and query it).

It's no work at all to go from regular language to context-free language, and a whole lot more work (parsing) to go the other way.
Post reply on HN