Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tracing: Structured logging, but better

andydote.co.uk

Post reply on HN
Print view View original

101–110 of 130 posts

Re: Tracing: Structured logging, but better

#101

Post by csomar »

zoogeny wrote: post #23

One thing about logging and tracing is the inevitable cost (in real money). I love observability probably more than most. And my initial reaction to this article is the obvious: why not both? In fact, I tend to think more in terms of "events" when writing both logs and tracing code. How that event is notified, stored, transmitted, etc. is in some ways divorced from the activity. I don't care if it is going to stdout,…

How does one break into the industry though? I worked in a project tangentially related and the problem is that sales were done by corporate sales man rather than on technicality. The companies buying the product didn't care because the people involved were making "deals". The company selling the product didn't care about making the product better because it was selling and having high AWS bills sounds like they were doing something (even though they were burning money).

Re: Tracing: Structured logging, but better

#102

Post by mikeshi42 »

_moog wrote: post #82
phillipcarter wrote: post #26

Earlier quoted context omitted.

FWIW part of the reason you're seeing that is, at least traditionally, APM companies rebranding as Observability companies stuffed trace data into metrics data stores, which becomes prohibitively expensive to query with custom tags/attributes/fields. Newer tools/companies have a different approach that makes cost far more predictable and generally lower. Luckily, some of the larger incumbents are also moving away fro…

> Newer tools/companies have a different approach that makes cost far more predictable and generally lower. What newer tools/companies are in this category? Any that you recommend?

I think we fit in that bucket [1] - open source, self-hostable, based on OpenTelemetry and backed by Clickhouse DB (columnar, not time-series).

Clickhouse gives users much greater flexibility in tradeoffs than either a time-series or inverted-index based store could offer (along with S3 support). There's nothing like a system that can balance high performance AND (usable) high cardinality.

[1] https://github.com/hyperdxio/hyperdx

disclaimer (in case anyone just skimmed): I'm one of the authors of HyperDX

Re: Tracing: Structured logging, but better

#103

Post by koliber »

Does this naive approach work for anyone to allow a log to be read like a trace:

1. At the start of a request, generate a globally unique traceId

2. Pass this traceId through the whole call stack.

3. Whenever logging, log the traceId as a parameter

Now you have a log with many of the plusses of a trace. The only additional cost to the log is the storage of the traceId on every message.

If you want to read a trace, search through your logs for "traceId: xyz123". If you use plain text storage you can grep. If you use some indexed storage, search for the key-value pair.

This way, you can retrieve something that looks like a trace from a log.

This does not solve all the issues named in the article. However, it is a decent tradeoff that I've used successfully in the past. Call it "poor man's tracing".

Re: Tracing: Structured logging, but better

#104

Post by FridgeSeal »

FridgeSeal wrote: post #71
crabbone wrote: post #25

> The second problem with writing logs to stdout Who on Earth does that? Logs are almost always written to stderr... In part to prevent other problems author is talking about (eg. mixing with the output generated by the application). I don't understand why this has to be either or... If you store the trace output somewhere you get a log... (let's call it "un-annotated" log, since trace won't have the human-readable m…

I do, as does everyone at my work? Along with basically everyone I’ve ever worked with, ever? Like, I develop cli apps, so like, what else would go to stdout that you suppose will interfere?

Can’t edit this now, but this is supposed to say “I don’t develop cli apps”

Re: Tracing: Structured logging, but better

#105

Post by hahn-kev »

zoogeny wrote: post #23

One thing about logging and tracing is the inevitable cost (in real money). I love observability probably more than most. And my initial reaction to this article is the obvious: why not both? In fact, I tend to think more in terms of "events" when writing both logs and tracing code. How that event is notified, stored, transmitted, etc. is in some ways divorced from the activity. I don't care if it is going to stdout,…

Why would you ever have lockin if you're using open telemetry?

Re: Tracing: Structured logging, but better

#106

Post by conradludgate »

lmm wrote: post #90
conradludgate wrote: post #81

Earlier quoted context omitted.

Are you running a debugger on a web service in production?

I wouldn't call it a "debugger", but plenty of people run an instrumentation agent like New Relic or AppDynamics that records tracing information on their production web services with little or even zero modification to their application code.

> zero modifications

Highly depends on the language or framework. I used to use NewRelic APM with Go and it required additional code to instrument.

All languages I have used in production require manual instrumentation. For that tracing and logging both work well in my experience.

Re: Tracing: Structured logging, but better

#107

Post by ryanklee »

ducharmdev wrote: post #85
ryanklee wrote: post #17

Earlier quoted context omitted.

I've never encountered this confusion anywhere, so I wouldn't ever think to dispel it. Which isn't to say that I disagree with the more general point that defining your terms is good thing. In any case, the post itself (which is not long) illustrates and marks out many of the differences.

I would guess that you're either not around junior engineers, or people are very good at hiding their confusion.

I wouldn't assert that the confusion is non-existent. But I think the audience for a post comparing technical differences between logging and tracing is unlikely a junior one.

But again, I do think the (brief) post marks out the differences throughout, so regardless, it still doesn't strike me as a problem here.

Re: Tracing: Structured logging, but better

#108

Post by Spivak »

rewmie wrote: post #57
ahoka wrote: post #56

Earlier quoted context omitted.

I think OP meant event sourcing.

> I think OP meant event sourcing. That is really besides the point. Logging and tracing have always been fundamentally event sourcing, but that never forced anyone ever at all to onboard onto freaking Kafka of all event streaming/messaging platforms. This blend of suggestion sounds an awful lot like resume driven development instead of actually putting together a logging service.

Hard disagree, Kafka is one of the simplest lowest maintenance tools for this with excellent language support and would probably be the first choice for anyone not paying $cloud_vendor for a managed durable queue.

The first step in building a reliable logging system is setting up a high write throughout highly available FIFOish durable storage. Once you have that everything else gets a lot easier.

* Once the log is committed to the durable queue that's it the application can move on secure the log isn't going to get lost.

* Multiple consumer groups can process the logs for different purposes, the usuals are one group for persisting the logs to a searchable index and one group for real time altering.

* Everything downstream from Kafka can be far less reliable because it's just a queue backup.

* You can fake more throughout then you actually have in your downstream processors because it just manifests as a lagging offset.

Re: Tracing: Structured logging, but better

#109

Post by gazpacho »

One big failing of OpenTelemetry's traces in particular is that attaching structured data to them is difficult. Most structured logs can be JSON which for all of it's faults most things can be serialized to JSON. OpenTelemetry's attributes on traces are much more limited, they don't even support a null/None value! I wish they just accepted JSON-like data, it'd make it much easier to always use traces.

Re: Tracing: Structured logging, but better

#110

Post by pondidum »

vkoskiv wrote: post #24

Nit to the author: 'rapala' seems like a mistranslation. It is a brand name of a company that makes fishing lures, as far as I can tell. It is not the Finnish word for "to bait", and is therefore only used to refer to a that particular brand. I'm not sure what the purpose of the text in parenthesis is here, but 'houkutella' would be the most apt translation in this case.

Thanks, I have fixed the definition in the post! Turns out its just company slang for bating, rather than Finnish slang!
Post reply on HN