Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Reducing logging cost by two orders of magnitude using CLP

uber.com

Post reply on HN
Print view View original

71–80 of 107 posts

Re: Reducing logging cost by two orders of magnitude using CLP

#71

Post by kevindong »

xwolfi wrote: post #58
kevindong wrote: post #16

Earlier quoted context omitted.

You can't realistically expect every log format to get a custom schema declared for it prior to deployment.

If you never intend to monitor them systematically, absolutely! If you're a bit serious you can at least impose date, time to the millisecond, pointer to the source of the log line, level, and a message. Let s be crazy and even say the message could have a structure too, but I can feel the weight of effort on your shoulders and say you ve already saved yourself the embarassement a colleague of mine faced when he real…

Sorry if I was ambiguous before. When I said "log format", I was referring to the message part of the log line. Standardized timestamp, line in the source code that emitted the log line, and level are the bare minimum for all logging.

Keeping the message part of the log line's format in sync with some external store is deviously difficult particularly when the interesting parts of the log are the dynamic portions that can take on multiple shapes.

Re: Reducing logging cost by two orders of magnitude using CLP

#72

Post by gcr »

pmarreck wrote: post #54
gdcohen wrote: post #14

Earlier quoted context omitted.

Does anyone have a simple explanation of how it structures the log data?

1) Use Zstandard with a generated dictionary kept separate from the data, but moreover: 2) Organize the log data into tables with columns, and then compress by column (so, each column has its own dictionary). This lets the compression algorithm perform optimally, since now all the similar data is right next to itself. (This reminds me of the Burrows-Wheeler transform, except much more straightforward, thanks to how s…

Does the user have to specify the "schema" (each unique log message type) manually? or is it learned automatically (a la gzip and friends)? I wasn't able to discover this from a cursory readthrough of the paper...

Re: Reducing logging cost by two orders of magnitude using CLP

#73

Post by cjalmeida »

taftster wrote: post #53

I'm not trying to flame bait here, but this whole article refutes the "Java is Dead" sentiment that seems to float around regularly among developers. This is a very complicated and sophisticated architecture that leverages the JVM to the hilt. The "big data" architecture that Java and the JVM ecosystem present is really something to be admired, and it can definitely move big data. I know that competition to this arch…

Of note, Java !== JVM. Spark and Flink, for instance, are written in Scala which is alive and well :).

My best effort in finding replacements of those tools that don't leverage the JVM:

HDFS: Any cloud object store like S3/AzBlob, really. In some workloads data locality provided by HDFS may be important. Alluxio can help here (but I cheat, it's a JVM product)

Spark: Different approach but you could use Dask, Ray, or dbt plus any SQL Analytical DB like Clickhouse. If you're in the cloud, and are not processing 10s TB at a time, spinning an ephemeral HUGE VM and using something in-memory like DuckDB, Polars or DataFrame.jl is much faster.

Yarn: Kubernetes Jobs. Period. At this point I don't see any advantage of Yarn, including running Spark workloads.

Hive: Maybe Clickhouse for some SQL-like experience. Faster but likely not at the same scale.

Storm/Flink/Cassandra: no clue.

My preferred "modern" FOSS stack (for many reasons) is Python based, with the occasional Julia/Rust thrown in. For a medium scale (ie. few TB daily ingestion), I would go with:

Kubernetes + Airflow + ad-hoc Python jobs + Polars + Huge ephemeral VMs.

Re: Reducing logging cost by two orders of magnitude using CLP

#74

Post by Cerium »

eterm wrote: post #50
otikik wrote: post #42

I really didn't know whether this was going to be an article about structuring sequential information or about a more efficient way to produce wood. Hacker news! I clicked, found out, and was dissapointed that this wasn't about wood. Maybe I should start that woodworking career change already.

Given that uber.com is prominently displayed in the title, I don't believe this charming and relatable little anecdote about title confusion.

Eh, I didn't notice Uber while I was considering if this was about automation of forestry operations. Though I did stop to consider if there is even enough waste in the industry to have potential gains of 100x. I doubt there is more than 2x available.

Re: Reducing logging cost by two orders of magnitude using CLP

#75

Post by layer8 »

Gh0stRAT wrote: post #63
gtowey wrote: post #21

Earlier quoted context omitted.

I like to say that when you try to make a "schemaless" database, you've just made 1000 different schemas instead.

Yeah, "Schemaless" is a total misnomer. You either have "schema-on-write" or "schema-on-read".

Schemaless means there’s no assurance that the stored data matches any consistent schema. You may try to apply a schema on read, but you don’t know if the data being read will match it.

Re: Reducing logging cost by two orders of magnitude using CLP

#76

Post by ofrzeta »

cjalmeida wrote: post #73
taftster wrote: post #53

I'm not trying to flame bait here, but this whole article refutes the "Java is Dead" sentiment that seems to float around regularly among developers. This is a very complicated and sophisticated architecture that leverages the JVM to the hilt. The "big data" architecture that Java and the JVM ecosystem present is really something to be admired, and it can definitely move big data. I know that competition to this arch…

Of note, Java !== JVM. Spark and Flink, for instance, are written in Scala which is alive and well :). My best effort in finding replacements of those tools that don't leverage the JVM: HDFS: Any cloud object store like S3/AzBlob, really. In some workloads data locality provided by HDFS may be important. Alluxio can help here (but I cheat, it's a JVM product) Spark: Different approach but you could use Dask, Ray, or…

There's ScyllaDB as a replacement for Cassandra. https://www.scylladb.com/

Re: Reducing logging cost by two orders of magnitude using CLP

#77

Post by mdaniel »

ofrzeta wrote: post #76
cjalmeida wrote: post #73

Earlier quoted context omitted.

Of note, Java !== JVM. Spark and Flink, for instance, are written in Scala which is alive and well :). My best effort in finding replacements of those tools that don't leverage the JVM: HDFS: Any cloud object store like S3/AzBlob, really. In some workloads data locality provided by HDFS may be important. Alluxio can help here (but I cheat, it's a JVM product) Spark: Different approach but you could use Dask, Ray, or…

There's ScyllaDB as a replacement for Cassandra. https://www.scylladb.com/

That's probably only true for extremely license-permissive shops:

https://github.com/apache/cassandra/blob/trunk/LICENSE.txt (Apache 2)

https://github.com/scylladb/scylladb/blob/master/LICENSE.AGP...

Re: Reducing logging cost by two orders of magnitude using CLP

#78

Post by mdaniel »

thethimble wrote: post #35
tomgs wrote: post #33

Disclaimer: I run Developer Relations for Lightrun. There is another way to tackle the problem for most normal, back-end applications: Dynamic Logging[0]. Instead of adding a large of amount of logs during development (and then having to deal with compressing and transforming them later) one can instead choose to only add the logs required at runtime. This is a workflow shift, and as such should be handled with care.…

Perhaps I’m misunderstanding but what happens if you’ve had a one-off production issue (job failed, etc) and you hadn’t dynamically logged the corresponding code? You can’t go back in time and enable logging for that failure right?

An alternative approach, IMHO, is to log all the things and just be judicious about expunging old stuff -- I believe the metrics community buys into this approach, too, storing high granularity captures for a week or whatever, and then rolling them up into larger aggregates for longer-term storage

I would also at least try a cluster-local log buffering system that forwards INFO and above as received, but buffers DEBUG and below, optionally allowing someone to uncork them if required, getting the "time traveling logging" you were describing. The risk, of course, is the more chains in that transmission flow the more opportunities for something to go sideways and take out all logs which would be :-(

Re: Reducing logging cost by two orders of magnitude using CLP

#79

Post by inkeddeveloper »

VectorLock wrote: post #45
marcosdumay wrote: post #38

Earlier quoted context omitted.

> There's nothing wrong with storing json in your database if the tradeoffs are clear and it's used in a sensible way. Of course. If there was, postgres wouldn't even support it. The GP's rant is usually thrown against people that default into json instead of thinking about it and maybe coming up with an adequate structure. There are way too many of those people.

Rigid structures and schemas are nice, but having document oriented data also has its advantages.

And so are document storage databases.

Re: Reducing logging cost by two orders of magnitude using CLP

#80

Post by mdaniel »

tomgs wrote: post #37
thethimble wrote: post #35

Earlier quoted context omitted.

Perhaps I’m misunderstanding but what happens if you’ve had a one-off production issue (job failed, etc) and you hadn’t dynamically logged the corresponding code? You can’t go back in time and enable logging for that failure right?

That would entail time-travelling and capturing that exact spot in the code, which is usually done by exception monitoring/handling products (plenty exist on the market). We're more after ongoing situations, where the issue is either hard to reproduce locally or requires very specific state - APIs returning wrong data, vague API 500 errors, application transactions issues, misbehaving caches, 3rd party library errors…

> which is usually done by exception monitoring/handling products (plenty exist on the market).

Only if one considers the bug/unexpected condition to be an exception; the only thing worse than nothing being an exception is everything being an exception

Post reply on HN