Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Reducing logging cost by two orders of magnitude using CLP

uber.com

Post reply on HN
Print view View original

91–100 of 107 posts

Re: Reducing logging cost by two orders of magnitude using CLP

#91

Post by tuatoru »

VectorLock wrote: post #45
marcosdumay wrote: post #38

Earlier quoted context omitted.

> There's nothing wrong with storing json in your database if the tradeoffs are clear and it's used in a sensible way. Of course. If there was, postgres wouldn't even support it. The GP's rant is usually thrown against people that default into json instead of thinking about it and maybe coming up with an adequate structure. There are way too many of those people.

Rigid structures and schemas are nice, but having document oriented data also has its advantages.

> having document oriented data also has its advantages.

As the VC said to the founder who wanted to do things the naive way: "well, I've seen quite a few people try that over the last few decades. It'll be interesting to see if you can make it work this time."

Re: Reducing logging cost by two orders of magnitude using CLP

#92

Post by genewitch »

hermanradtke wrote: post #70
tomgs wrote: post #65

Earlier quoted context omitted.

Pricing is here: https://lightrun.com/pricing

I don’t consider Free for one agent and “contact us” for everything else to be pricing.

logging is one of those things that gets complex in a hurry. if i was starting a log processing company i'd also do this, since pure bytes isn't a good metric. A really long error chain (like java gives) is "one error", but is that stored as a single "error" or multiple lines and parsed out later?

I really stopped paying attention to logging around logstash era for enterprise solutions, so my knowledge is woefully out of date, but based on how crazy my python services logfiles get for a single instance, i don't know that much has really changed.

Re: Reducing logging cost by two orders of magnitude using CLP

#93

Post by Rexxar »

SkeuomorphicBee wrote: post #12

"Page not found" Apparently the Uber site noticed I'm not in the USA and automatically redirects to a localized version, which doesn't exist. If their web-development capabilities are any indication I'll skip their development tips.

I have the same problem but the redirect only fail when I use Firefox.

Re: Reducing logging cost by two orders of magnitude using CLP

#94

Post by shrubble »

sa46 wrote: post #68
shrubble wrote: post #61

This is basically sysadmin 101, however. Compressing logs has been a thing since the mid-1990s. Minimizing writes to disk, or setting up a way to coalesce the writes, has also been around for as long as we have had disk drives. If you don't have enough RAM on your system to buffer the writes so that more of the writes get turned into sequential writes, your disk performance will suffer - this too has been known since…

Sysadmin 101 doesn't involve separating the dynamic portions of similar, but unstructured log lines to dramatically improve compression and search performance. > Zstandard or Gzip do not allow gaps in the repetitive pattern; therefore when a log type is interleaved by variable values, they can only identify the multiple substrings of the log type as repetitive.

A sysadmin would use the logging facility (if traditional syslog) or simply awk/sed to process the logs into different files that are similar to each other (such as different levels of INFO/WARN/ERROR); then, increase the size of the DEFLATE dictionary used for compression until you get better compression.

See for instance this discussion of creating a custom DEFLATE dictionary: https://blog.cloudflare.com/improving-compression-with-prese...

Re: Reducing logging cost by two orders of magnitude using CLP

#95

Post by AtlasBarfed »

npalli wrote: post #56
taftster wrote: post #53

I'm not trying to flame bait here, but this whole article refutes the "Java is Dead" sentiment that seems to float around regularly among developers. This is a very complicated and sophisticated architecture that leverages the JVM to the hilt. The "big data" architecture that Java and the JVM ecosystem present is really something to be admired, and it can definitely move big data. I know that competition to this arch…

I didn't read the article that way. FTA, the sense is Java is not dead in the same sense COBOL is not dead, that is "legacy" technology that you have now work around because it is too costly to operate and maintain. Ironically, from this article the two main technical solves for the issues with their whole JVM setup are CLP (which is the main article) and moving to Clickhouse for non-Spark logs both of which are writ…

"Java is the new COBOL" has always been either a glaring sign of idiocy/ignorance or a bad joke signifying... idiocy/ignorance.

COBOL is exotic syntax and runs on fringe/exotic hardware (mainframes, minicomps, IBM iron).

Java is a c-like syntax that runs everywhere people are shoehorning in Go and Node.JS. Syntax arguments are bikeshedding, but it was a "step forward" for non-systems coding from C and has fundamental design, architectural, breadth of library, interop, modernization, and familiarity advantages over COBOL.

Go is a syntax power stepback, with possibly some GC advantages, and Javascript even with Typescript is still a messed up ecosystem with worse GC and performance issues.

One thing that was interesting was watching the Ruby on Rails stack explode in complexity to encompass an acronym soup nearly as bad a Java as the years moved forward and it matured. Java isn't as complex an ecosystem as it is due to any failings or language failures. It simply has to be as a mature ecosystem.

Syntax complaints I'll listen too, after all I do all my JVM programming in Groovy. But if you complain about java syntax, why would you think Go is "better"?

I think a meta-language will emerge that will have Rust power and checking underlying it but a lot simpler, kind of like elixir and erlang, or typescript and javascript, or, well, Groovy and Java.

Re: Reducing logging cost by two orders of magnitude using CLP

#96

Post by paulmd »

foobarian wrote: post #67
taftster wrote: post #53

I'm not trying to flame bait here, but this whole article refutes the "Java is Dead" sentiment that seems to float around regularly among developers. This is a very complicated and sophisticated architecture that leverages the JVM to the hilt. The "big data" architecture that Java and the JVM ecosystem present is really something to be admired, and it can definitely move big data. I know that competition to this arch…

Not answering your primary question, I know. But I wonder where you are getting the "Java is Dead" sentiment - I am not getting it at all in my (web/enterprisey) circle, if anything there is a lot of excitement due to new LTS versions and other JVM languages like Kotlin. And I am also finding a lot of gratitude for the language not changing in drastic ways (can you imagine a Python 2->3 like transition?) despite the…

Java has never been more alive really. All the other JVM languages just strengthen and retrench JVM, it all deploys the same. And Java itself has really come a long long ways since the JDK7 days.

Re: Reducing logging cost by two orders of magnitude using CLP

#97

Post by tarasglek »

trueleo wrote: post #36
bcjordan wrote: post #3

Always bugged me that highly repetitive logs take up so much space! I'm curious, are there any managed services / simple to use setups to take advantage of something like this for massive log storage and search? (Most hosted log aggregators I've looked at charge by the raw text GB processed)

Check out https://github.com/parseablehq/parseable ... we are building a log storage and analysis platform in rust. Columnar format helps a lot in reducing overall size but then you have little computational overhead to deal with conversion and compression. This trade off will be there but we are discovering ways to minimise it with rust

It doesn't seem like your solution achieves columnar breakdowns for unstructured parts of the log. Eg they will basically reverse engineer printfs, you don't. Misleading claim of being similar

Re: Reducing logging cost by two orders of magnitude using CLP

#99

Post by b0ti »

xwolfi wrote: post #34
dewey wrote: post #8

Earlier quoted context omitted.

There's nothing wrong with storing json in your database if the tradeoffs are clear and it's used in a sensible way. Having structured data and an additional json/jsonb column where it makes sense can be very powerful. There's a reason every new release of Postgres improves on the performance and features available for the json data type. ( https://www.postgresql.org/docs/9.5/functions-json.html )

It cant be. Json has a huge structural problem: it's an ASCII representation of a schema+value list, where the schema is repeated with each value. It improved on xml because it doesn't repeat the schema twice, at least... It's nonsensical most of the time: do a table, transform values out of the db or in the consumer. The reason postgres does it is because lazy developpers overused the json columns and then got fucke…

Couldn't agree more. Not to mention timestamps that JSON simply doesn't handle and is essential for event data. The raijin database has an clever approach to solving the schema rigidity problem: https://raijin.co

Re: Reducing logging cost by two orders of magnitude using CLP

#100

Post by [deleted] »

kazinator wrote: post #29
hobs wrote: post #2

This just in, Uber rediscovers what all us database people already knew, structured data is usually way easier to compress and store and index and query than unstructured blobs of text, which is why we kept telling you to stop storing json in your databases.

Maybe people do things like that because: - their application parses and generates JSON already, so it's low-effort. - the JSON can have various shapes: database records generally don't do that. - even if it has the same shape, it can change over time; they don't want to deal with the insane hassle of upgrade-time DB schema changes in existing installations The alternative to JSON-in-DB is to have a persistent object…

[deleted]
Post reply on HN