Thank you for helping me structure what I want, yes, exactly. (And noted re: your PoC, ok!)
> You might want to see all events of the internals of two different components in the tree structure.
> Need to process log lines into a graph and then provide quick navigation for them.
Yes, looks like it, indeed. Yeah.
If interested, this is how I'd hastily summarise my problem statement - it is I think a rather generic recurring problem and analysis pattern, not unique to any company (context: goal is to drive through this initiative as part of overall software architecture improvement needs, and bringing business maturity (as per CMM model) up in our particular case):
- need to introduce and enforce structured event hierarchy (think: each high level business transaction (e.g. user signs document) is a parent which retains all related child events across all layers of abstraction (ideally down to DB connection and file handles, internal API endpoints hit, block storage i/o errors, etc.)
- problem: while for own software Backend-generated events we can enforce and manage this, some infra pieces emit raw text syslog (sometimes even syslog message portion (log format) cannot be fully controlled) and do not contain cross-component technical identifiers for determining relations to events
- there are solutions including ML solutions (of course)
- there exist interesting papers to lose productivity and/or conduct deeper research
- what do, how to balance, how to incrementally define, introduce into culture, incrementally realistically establish this (etc.)
Some interesting papers for approaching the complicated part (assuming defining event ontology and hierarchy and overall semantics is the easy part) - the "relating unstructured logs to events and embedding them into graphs, and aligning and training everyone to analyse and work with "event trees" across multiple teams (but yes, ops, integration support, etc. - I mean is that not the dream, right...) do exist[1][2][3][4][5], and some interesting thoughts and many lofty ambitious questions[6]. Trying to gain insight into best practices here. Meanwhile, maybe links will look interesting :) (e.g. that 1849-2023 IEEE standard - just found it today...)
[1] LogTree: A Framework for Generating System Events from Raw Textual Logs https://www.researchgate.net/publication/220766390_LogTree_A... (https://ltangt.github.io/papers/icdm2010-logtree.pdf)
[2] Structural Event Detection from Log Messages https://faculty.ist.psu.edu/jessieli/Publications/2017-KDD-l... (this one may look bewildering at first but the math eqs are actually ~legible / straightforward from quick glance, even; also, has nice coverage of related work) (overall nice, might look smarter than appears, but not sure; see MCMC approach etc. in section 5.)
[3] golang package https://pkg.go.dev/source.monogon.dev/metropolis/pkg/logtree - why i mention - see KLogParser func (https://pkg.go.dev/source.monogon.dev/metropolis/pkg/logtree...) (but also see limitations) (did not review in detail, maybe nothing useful)
[4] 1849-2023 - IEEE Standard for eXtensible Event Stream (XES) for Achieving Interoperability in Event Logs and Event Streams https://www.xes-standard.org/
[5] related to above (supports standard) https://pm4py.fit.fraunhofer.de/documentation -> see example of ontology: Object-Centric Event Logs
[6] a structural approach to the problem of structure... use existing frameworks and languages like BPML to create or use existing event type ontologies from BPMN models; use a proper structured thought-through ontology for this, and then sort out that pesky "relating low level infra raw syslogs with none/few TIDs to some node on a graph" problem
Anyway, thanks for making through my rant! Writing it in part because on the off chance someone bites and has insight - I'd be very interested to hear more; and partially just to share those links above because those may tickle your (Sam's fancy) :) thanks for reading!