Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Log messages are mostly for the people operating your software

utcc.utoronto.ca

Post reply on HN
Print view View original

51–60 of 83 posts

Re: Log messages are mostly for the people operating your software

#51

Post by john_strinlai »

dexwiz wrote: post #4
hk__2 wrote: post #3

Earlier quoted context omitted.

> Most users don't even read error messages, never mind logs. Yes, see all the questions on StackOverflow with people posting their error message without reading it, like “I got an error that says ‘fail! please install package xyz!’, what should I do?!?”.

That question is more likely how do I install, not what to install.

on friday i got 2 calls saying "my phone is no longer showing me my emails, please fix" when the error message they received was roughly "please reenter your password to continue using outlook".

on wednesday i got a call saying "the CRM wont let me input this note, please fix" when the error message was "you have included an invalid character, '□' found in description. remove invalid characters and resubmit".

Re: Log messages are mostly for the people operating your software

#53

Post by pdntspa »

bonoboTP wrote: post #30
LgWoodenBadger wrote: post #7

All software should provide something meaningful for anybody to diagnose, if they’re inclined to. It’s particularly bad in the (Apple) mobile ecosystem, including AppleTV. I have AdGuard Home but one of my spouse’s streaming services wouldn’t work. “There was a problem.” Gee thanks. Eventually figured out that I had to unblock a few hosts so it would work. Only found which ones by googling and finding some other poor…

They don't want tinkering or tinkerers. Apple is all about walled-off, locked-down, black box, just-works (when it does) etc. It's supposed to seem like magic. You're not supposed to tinker with magic, it makes it pedestrian. Apple as a brand is a lifestyle, a feeling. The slick, polished brand. Remember "I'm a Mac, and I'm a PC"? PC is where you tinker, and there is screws and nuts and bolts and jargon and troublesh…

I wouldn't confuse Steve Jobs-era Apple with what it is now.

Re: Log messages are mostly for the people operating your software

#54

Post by vlovich123 »

hinkley wrote: post #38

For years now I’ve been pushing for moving of all non actionable error messages and all aggregate-actionable error messages into telemetry data instead. Not the least of which because log processing SaaS companies seem to be overcharging for their services even versus hosted Grafana services, and really many of us could do away with the rent seeking entirely. The computational complexity of finding meaning in log fil…

How do you handle the problem that telemetry is generally incapable of capturing temporal context?

Re: Log messages are mostly for the people operating your software

#55

Post by not_kurt_godel »

bonoboTP wrote: post #30
LgWoodenBadger wrote: post #7

All software should provide something meaningful for anybody to diagnose, if they’re inclined to. It’s particularly bad in the (Apple) mobile ecosystem, including AppleTV. I have AdGuard Home but one of my spouse’s streaming services wouldn’t work. “There was a problem.” Gee thanks. Eventually figured out that I had to unblock a few hosts so it would work. Only found which ones by googling and finding some other poor…

They don't want tinkering or tinkerers. Apple is all about walled-off, locked-down, black box, just-works (when it does) etc. It's supposed to seem like magic. You're not supposed to tinker with magic, it makes it pedestrian. Apple as a brand is a lifestyle, a feeling. The slick, polished brand. Remember "I'm a Mac, and I'm a PC"? PC is where you tinker, and there is screws and nuts and bolts and jargon and troublesh…

You're baffled because you appear to be uninformed and/or willfully ignorant. macOS is Unix-based and 90% functionally equivalent to Linux for software development and tinkering purposes. iOS, while less customizable than Android, is overall very good software for a phone. Apple hardware is superior across the board, especially for durability.

Meanwhile, I'm baffled why any techie would voluntarily use an OS that force-enables telemetry and advertising. The fight for privacy and ad-free experiences is hard enough without your OS fundamentally working against you.

Re: Log messages are mostly for the people operating your software

#56

Post by [deleted] »

DharmaPolice wrote: post #41
bonoboTP wrote: post #30

Earlier quoted context omitted.

They don't want tinkering or tinkerers. Apple is all about walled-off, locked-down, black box, just-works (when it does) etc. It's supposed to seem like magic. You're not supposed to tinker with magic, it makes it pedestrian. Apple as a brand is a lifestyle, a feeling. The slick, polished brand. Remember "I'm a Mac, and I'm a PC"? PC is where you tinker, and there is screws and nuts and bolts and jargon and troublesh…

I totally agree but you can attribute a lot of the Apple worship to Microsoft and their OEM partners making PC laptops an often miserable experience.

[deleted]

Re: Log messages are mostly for the people operating your software

#57

Post by hinkley »

xahrepap wrote: post #49
hinkley wrote: post #38

For years now I’ve been pushing for moving of all non actionable error messages and all aggregate-actionable error messages into telemetry data instead. Not the least of which because log processing SaaS companies seem to be overcharging for their services even versus hosted Grafana services, and really many of us could do away with the rent seeking entirely. The computational complexity of finding meaning in log fil…

Honest question, how do you handle high cardinality data points? Reference to where my brain is at: https://www.robustperception.io/cardinality-is-key/ I feel like splunk’s business model favors a healthy system and gives major disadvantages to an unhealthy one. What I mean in an example: when the system is unhealthy, I know it because all my splunk queries get queued up because everyone is slamming it with queries.…

This gets even worse if you have a language with one process per CPU as you can get clobbering other values on the same instance if you don't add fields to uniquely identify them.

We got a lot of pushback when migrating our telemetry to AWS after initially being told to just move it when they saw how OTEL amplified data points and cardinality versus our old StatsD data.

You probably need less cardinality than you think, and there are a mix of stats that work fine with less frequent polling, while others like heap usage are terrible if you use 20 or 30 second intervals. Our Pareto frontier was to reduce the sampling rate of most stats and push per-process things like heap usage into histograms.

An aggregator per box can drop a couple of tags before sending them upstream which can help considerably with the number of unique values. (eg, instanceID=[0..31] isn't that useful outside of the box)

Re: Log messages are mostly for the people operating your software

#58

Post by hinkley »

vlovich123 wrote: post #54
hinkley wrote: post #38

For years now I’ve been pushing for moving of all non actionable error messages and all aggregate-actionable error messages into telemetry data instead. Not the least of which because log processing SaaS companies seem to be overcharging for their services even versus hosted Grafana services, and really many of us could do away with the rent seeking entirely. The computational complexity of finding meaning in log fil…

How do you handle the problem that telemetry is generally incapable of capturing temporal context?

That is the $65k question and unfortunately I don't have a pat answer for that yet. I probably need to see more types of projects instead of more time on fewer projects which is where I'm at.

But I can give you a partial picture.

You're going to end up with multiple dashboards with duplicate charts on them because you're showing correlation between two charts via proximity. Especially charts that are in the same column in row n±1 or vice versa. You're trying to show whether correlation is likely to be causation or not. Grafana has a setting that can show the crosshairs on all graphs at the same time, but they need to be in the same viewport for the user to see them. Generally, for instance, error rates and request rates are proportional to each other, unless a spike in error rates is being for instance triggered by web crawlers who are now hitting you with 300 req/s each whereas they normally are sending you 50. The difference in the slope of the lines can tell you why an alert fired or that it's about to. So I let previous RCAs inform whether two graphs need to be swapped because we missed a pattern that spanned a viewport. And sometimes after you fix tech debt, the correlation between two charts goes up or way down. So what was best in May not be best come November.

There's a reason my third monitor is in portrait mode, and why that monitor is the first one I see when I come back to my desk after being AFK. I could fit 2 dashboards and group chat all on one monitor. One dashboard showed overall request rate and latency data, the other showed per-node stats like load and memory. That one got a little trickier when we started doing autoscaling. The next most common dashboard which we would check at intervals showed per-service tail latencies versus request rates. You'd check that one every couple of hours, any time there was a weird pattern on the other two, or any time you were fiddling with feature toggles.

From there things balkanized a bit. We had a few dashboards that two or three of us liked and the rest avoided.

Re: Log messages are mostly for the people operating your software

#59

Post by ting0 »

not_kurt_godel wrote: post #55
bonoboTP wrote: post #30

Earlier quoted context omitted.

They don't want tinkering or tinkerers. Apple is all about walled-off, locked-down, black box, just-works (when it does) etc. It's supposed to seem like magic. You're not supposed to tinker with magic, it makes it pedestrian. Apple as a brand is a lifestyle, a feeling. The slick, polished brand. Remember "I'm a Mac, and I'm a PC"? PC is where you tinker, and there is screws and nuts and bolts and jargon and troublesh…

You're baffled because you appear to be uninformed and/or willfully ignorant. macOS is Unix-based and 90% functionally equivalent to Linux for software development and tinkering purposes. iOS, while less customizable than Android, is overall very good software for a phone. Apple hardware is superior across the board, especially for durability. Meanwhile, I'm baffled why any techie would voluntarily use an OS that for…

He's uninformed? I assume you have a jailbroken Apple iPhone then?

Re: Log messages are mostly for the people operating your software

#60

Post by mixmastamyk »

not_kurt_godel wrote: post #55
bonoboTP wrote: post #30

Earlier quoted context omitted.

They don't want tinkering or tinkerers. Apple is all about walled-off, locked-down, black box, just-works (when it does) etc. It's supposed to seem like magic. You're not supposed to tinker with magic, it makes it pedestrian. Apple as a brand is a lifestyle, a feeling. The slick, polished brand. Remember "I'm a Mac, and I'm a PC"? PC is where you tinker, and there is screws and nuts and bolts and jargon and troublesh…

You're baffled because you appear to be uninformed and/or willfully ignorant. macOS is Unix-based and 90% functionally equivalent to Linux for software development and tinkering purposes. iOS, while less customizable than Android, is overall very good software for a phone. Apple hardware is superior across the board, especially for durability. Meanwhile, I'm baffled why any techie would voluntarily use an OS that for…

Apple sends tens of megabytes of telemetry from first network connection and regularly:

https://sneak.berlin/20210202/macos-11.2-network-privacy/

None of this able to be turned off, the boot volume is read-only. Can only be deactivated by jumping through hoops.

Post reply on HN