61–70 of 74 posts
Has Matt read any prior art in this field? https://research.google/pubs/monarch-googles-planet-scale-in...
Earlier quoted context omitted.
The thing about telemetry data is it's extremely repetitive. Take for example a CLF[1] log line: 127.0.0.1 user-identifier frank [10/Oct/2000:13:55:36 -0700] "GET /apache_pb.gif HTTP/1.0" 200 2326 As written this is 99 bytes (792 bits), but how much information is actually in it? We have an IP address which is taking up 9 bytes but only needs at most 4 (fewer in cases like this where two of the bytes are zero if we e…
My hunch is that JSON using a custom compression dictionary with zlib (see zdict argument to https://docs.python.org/3/library/zlib.html#zlib.compressobj ) or zstandard would get you most of the benefit while still letting you interact with existing JSON tools. I've not put the work in to prove that to myself though!
Have a look at the description and performance of a non-toy time series database published 10 years ago:
https://www.vldb.org/pvldb/vol8/p1816-teller.pdf
Convenience of text and json is an argument, but performance??
Earlier quoted context omitted.
if you don't have metrics for cpu/memory/storage how do you know when to scale the app, or when you are at limit of the storage? i feel like you have never touched servers/backend in anything more than simple projects (or at all). with full storage/memory there could be an issue that you won't be able to ssh to the server, so it speaks about your knowledge in this matter. collecting user-identified telemetry is debat…
> if you don't have metrics for cpu/memory/storage how do you know when to scale the app When the business metrics start to fail. You don't need constant metics on storage, you can poll it every so often. If your app is constrained by CPU or RAM, then the business metrics will reflect that, and then you can turn on collection of those metrics to identify the problem. > i feel like you have never touched servers/backe…
…but why incur that round trip on my feedback loop? Having those metrics on doesn’t cost me much.
This feels potentially like the perspective of a large organisation with both mature monitoring systems and quite steady state user base activity (through scale). When I have a customer who had an issue yesterday because they had an unusual workload that won’t be repeated often, I can’t afford not to have had the basic metrics turned on, in case they point us in the right direction.
Earlier quoted context omitted.
if you don't have metrics for cpu/memory/storage how do you know when to scale the app, or when you are at limit of the storage? i feel like you have never touched servers/backend in anything more than simple projects (or at all). with full storage/memory there could be an issue that you won't be able to ssh to the server, so it speaks about your knowledge in this matter. collecting user-identified telemetry is debat…
> if you don't have metrics for cpu/memory/storage how do you know when to scale the app When the business metrics start to fail. You don't need constant metics on storage, you can poll it every so often. If your app is constrained by CPU or RAM, then the business metrics will reflect that, and then you can turn on collection of those metrics to identify the problem. > i feel like you have never touched servers/backe…
After having annoyed how many users and lost how much revenue? Having metrics to identify brewing problems before issues start to arise (be they on arriving CPU, memory, disk, network constraints or increasing network latency which will soon but not yet show up in the business metrics) is valuable.
> I ran all of ops for reddit for four years and headed up SRE at Netflix, so I have some experience in large scale systems. Not that it should matter.
I have a hard time believing at either of those it was acceptable to have a problem ongoing for days without any idea what's happening because logs and metrics weren't enabled in the first place.
Earlier quoted context omitted.
when storage is full, and you don't know about that, you can't release anything to enable the logs in first place.
You can poll storage periodically though, you don't need to keep a constant metrics stream of where it's at. Also you can set up each machine to alert when it's own storage fills up. Also, as your storage hits 97%+, you'll probably start seeing effects in your business metrics, and then you can look into it.
Unless you want to be able to have trends over time, either for capacity planning (needing to order more storage in case of bare metal, or planning costs ahead) or to correlate with other things (storage consumption is growing twice as fast since deployment X, did we change something there?).
You don't need to have 1s granularity metrics on storage consumption, but having none is just stupid levels of fake "optimisation" that will cost you more in the long run.
Earlier quoted context omitted.
where you worked doesn't matter to me very much, when what are you saying contradicts what you probably did ("experience in large scale systems"), also it sounds like argument from authority. not having cpu/mem/hdd metrics is just plain bogus and sounds like fantasy world, where everything works like we expect it to work, and there is no bugs at all. ridiculous
> i feel like you have never touched servers/backend in anything more than simple projects (or at all) I feel like if you are going to go out on a limb and call someone's expertise into question... > I ran all of ops for reddit for four years and headed up SRE at Netflix And they provide excellent credentials which you failed to check... > where you worked doesn't matter to me very much You can't just weasel out of i…
that's logical fallacy, you can work in any place on earth and still be wrong in the subject.
> You can't just weasel out of it by pretending like you didn't start the interaction by calling someone's expertise into question.
why? if his take is bad, then his job or experience doesn't change the outcome. i'm not an expert by any means, but things that hes saying just contradict everything that is standard practice and my own experience. based on that i'm able to say that he doesn't know what he's saying/proposing, and using his "excellent credentials" just make things worse, as it shows that he doesn't have an argument, just wishful thinking
Earlier quoted context omitted.
where you worked doesn't matter to me very much, when what are you saying contradicts what you probably did ("experience in large scale systems"), also it sounds like argument from authority. not having cpu/mem/hdd metrics is just plain bogus and sounds like fantasy world, where everything works like we expect it to work, and there is no bugs at all. ridiculous
You question his competence. > i feel like you have never touched servers/backend in anything more than simple projects (or at all). with full storage/memory there could be an issue that you won't be able to ssh to the server, so it speaks about your knowledge in this matter. He was answering that. If instead of dismissing someone outright and question their competence, you had raised specific concerns, this would ha…
he first said that we don't need to monitor anything, just enable debugging when "business metrics" are failing, and then he changed his stance to "polling from time to time". that's just shows that his first take wasn't thoughtful, so I assumed that he never worked in "the field" or worked on smaller projects, as nobody that worked in bigger projects would say that "we don't need CPU/mem/hdd metrics". it's not like hes proposing something novel, that just ridiculous take that needs to be called out
Earlier quoted context omitted.
I think that you are confusing real-time metrics, streamed with very high precision (below 1s) and metrics that are simply polled every N time (most use-cases). real-time, high precision metrics aren't necessary. when you say that you don't need metrics and then say that you can poll metrics periodically, you are contradicting yourself.
I'm not contradicting myself. I'm saying you just poll for storage, you don't store the results. My entire thesis is that those metrics aren't worth storing .
Earlier quoted context omitted.
Strongly disagree. Having stored telemetry has helped me debug so many things. Forever is probably too much, but keeping a month or so is totally sane.
Why kind of things did you debug with CPU/Memory/Storage telemetry that you couldn't have debugged by only turning those things on after you knew there was a problem?
This service is using 80% CPU, that seems a bit high... but is it always this high? Looks like it spiked within the last hour. But wait, it does that every Monday at 9 am, so probably a red herring.
This cache has a hit ratio of 60%... is that good? A bit low? Actually it's suspiciously high compared to last week - looks like a lot of people aren't getting a personalised feed.
Metrics are incredibly cheap to keep around for the value you get from a good operational dashboard, despite what Datadog/Amazon/Grafana Cloud tells you. It's just the most egregiously overpriced data you can buy since 20 cent text messages.
A good start is to set up VictoriaMetrics with some collectors and set retention to 14 days.
Earlier quoted context omitted.
> i feel like you have never touched servers/backend in anything more than simple projects (or at all) I feel like if you are going to go out on a limb and call someone's expertise into question... > I ran all of ops for reddit for four years and headed up SRE at Netflix And they provide excellent credentials which you failed to check... > where you worked doesn't matter to me very much You can't just weasel out of i…
> And they provide excellent credentials which you failed to check... that's logical fallacy, you can work in any place on earth and still be wrong in the subject. > You can't just weasel out of it by pretending like you didn't start the interaction by calling someone's expertise into question. why? if his take is bad, then his job or experience doesn't change the outcome. i'm not an expert by any means, but things t…