61–70 of 107 posts
Compressing logs has been a thing since the mid-1990s.
Minimizing writes to disk, or setting up a way to coalesce the writes, has also been around for as long as we have had disk drives. If you don't have enough RAM on your system to buffer the writes so that more of the writes get turned into sequential writes, your disk performance will suffer - this too has been known since the 1990s.
Earlier quoted context omitted.
I didn't read the article that way. FTA, the sense is Java is not dead in the same sense COBOL is not dead, that is "legacy" technology that you have now work around because it is too costly to operate and maintain. Ironically, from this article the two main technical solves for the issues with their whole JVM setup are CLP (which is the main article) and moving to Clickhouse for non-Spark logs both of which are writ…
Just to probe. COBOL doesn't have many (if any) updates to it, though. And there are no big data architectures being built around it. Equating "Java is Dead" to the same meaning as "COBOL is Dead" doesn't seem like a legitimate comparison. But I do get your points and don't necessarily disagree with them. I just don't see this as "legacy" technology, but maybe more like "mature"?
Earlier quoted context omitted.
Word! Storing JSON is so often the most direct and explicit way of accruing technical debt: "We don't really know what structure the data we'll get should have, just specify that it's going to be JSON"...
I like to say that when you try to make a "schemaless" database, you've just made 1000 different schemas instead.
Disclaimer: I run Developer Relations for Lightrun. There is another way to tackle the problem for most normal, back-end applications: Dynamic Logging[0]. Instead of adding a large of amount of logs during development (and then having to deal with compressing and transforming them later) one can instead choose to only add the logs required at runtime. This is a workflow shift, and as such should be handled with care.…
Sounds pretty cool. How much?
Disclaimer: I run Developer Relations for Lightrun. There is another way to tackle the problem for most normal, back-end applications: Dynamic Logging[0]. Instead of adding a large of amount of logs during development (and then having to deal with compressing and transforming them later) one can instead choose to only add the logs required at runtime. This is a workflow shift, and as such should be handled with care.…
I'm glad someone put a name on the concept I've been advocating for a decade. Thank you! It's something we added at Netflix when we realized our logging costs were out of control. We had a dashboard where you could flip on certain logging only as needed.
I tried finding you on twitter but no go since DMs are closed.
Would be happy to pick your brain about the topic - tom@granot.dev is where I’m at if you have the time!
I'm not trying to flame bait here, but this whole article refutes the "Java is Dead" sentiment that seems to float around regularly among developers. This is a very complicated and sophisticated architecture that leverages the JVM to the hilt. The "big data" architecture that Java and the JVM ecosystem present is really something to be admired, and it can definitely move big data. I know that competition to this arch…
This is basically sysadmin 101, however. Compressing logs has been a thing since the mid-1990s. Minimizing writes to disk, or setting up a way to coalesce the writes, has also been around for as long as we have had disk drives. If you don't have enough RAM on your system to buffer the writes so that more of the writes get turned into sequential writes, your disk performance will suffer - this too has been known since…
> Zstandard or Gzip do not allow gaps in the repetitive pattern; therefore when a log type is interleaved by variable values, they can only identify the multiple substrings of the log type as repetitive.
"Page not found" Apparently the Uber site noticed I'm not in the USA and automatically redirects to a localized version, which doesn't exist. If their web-development capabilities are any indication I'll skip their development tips.