91–100 of 109 posts
Earlier quoted context omitted.
Is requiring someone to learn a programming language worse than requiring someone to learn a custom DSL? Seems a strange assertion.
Clojure is a bigger language than small custom DSL.
Greenspun's 10th rule? http://c2.com/cgi/wiki?GreenspunsTenthRuleOfProgramming
Earlier quoted context omitted.
The parts of Clojure required to write a config file are probably roughly the same size as a custom DSL though. Maps and lists are gonna be maps and lists...
Except for Clojure's data model. You'll still need to understand what apostrophe ("'") does and what keyword (":foo") is.
It's a good thing that it's Clojure and not some freakish turing-complete XML-based configuration file.
Anyone use http://prometheus.io/ ? wdyt
The lack of a plugin system, and the reliance on HTTP to collect, means writing small collectors is a pain. You can't run 23 different daemons, each on their own port, to collect stats from things like PostgreSQL stats or RabbitMQ.
We opted for the "text directory" way, where we populate a directory with .prom files that node_exporter automatically picks up. It's not ideal: It means a whole bunch of collectors run via Cron jobs, which themselves need to be monitored; it means if we remove a collector, we also have to clean up its .prom files; and in the end it meant we had to invent our own little plugin system in order to avoid writing a lot of boilerplate code needed to manage the different collectors we use. We'd love to share our collectors, but due to that last point, our collectors are less reusable than we'd like.
Prometheus itself has been quite stable, but it still has some rough edges:
* If anything goes wrong with its database files, it tends to just crash, and the only way out is to wipe the entire database (e.g., see [1]).
* There's no way to do snapshots of the database.
* The team is rather cavalier about backwards compatibility. We've experienced at least one version upgrade where they changed the database format and didn't provide any upgrade tools, so people were forced to start their metrics history from scratch. I know that it's pre-1.0, but still, they knew perfectly well that people were running it in production. The alert manager was also written from scratch recently, with a whole new config format. With several releases, every tool has had its command line flags changed ever so subtly, too.
* The lack of packages (Debian/Ubuntu in our case) is also problematic. Fortunately I've got a script now that grabs a release and bundles a .deb from it, but I'd vastly preferred real releases.
* No syslog support is not acceptable in this day and age. Our Upstart scripts spawns a "logger" subprocess to catch stderr. Not everyone is running under Docker.
Evaluated it and a couple of others. Chose Bosun, never looked back, it's probably the only system with flexible and concise DSL for evaluating alerts (somewhat similar to R in spirit). Riemann rocks, just not as monitoring system. Bosun link: http://bosun.org/
Anyone use http://prometheus.io/ ? wdyt
We're using Prometheus. Very happy, though the collection side of things aren't as smooth as we'd like it. The lack of a plugin system, and the reliance on HTTP to collect, means writing small collectors is a pain. You can't run 23 different daemons, each on their own port, to collect stats from things like PostgreSQL stats or RabbitMQ. We opted for the "text directory" way, where we populate a directory with .prom f…
We have many ways to plugin to Prometheus across the ecosystem, the textfile collector you're using is one of them.
> You can't run 23 different daemons, each on their own port, to collect stats from things like PostgreSQL stats or RabbitMQ.
There's no fundamental challenge with this approach. If you've got good basic infrastructure, particularly configuration management, the rollout of each should be a small operational task. If it's a major challenge, then your problem probably isn't with the Prometheus architecture.
> it means if we remove a collector, we also have to clean up its .prom files
There's several problems arising from this approach, this is one of them. You can also expect odd artifacts in graphs.
The textfile collector is only intended for machine-level metrics, by putting service level metrics in there you're missing out on a big win of Prometheus by thinking in terms of machines rather than services.
Fighting against the architecture means you're not getting the maximum benefits from Prometheus, this would be easier with exporters and service discovery.
> which themselves need to be monitored
Are you aware that the node exporter exports the mtime of all the textfile collector files? That's there to make monitoring of them easier.
> If anything goes wrong with its database files, it tends to just crash, and the only way out is to wipe the entire database
As far as we're aware, the only way that happens is if you run out of disk space. If you've evidence otherwise please let us know, so we can prioritize accordingly.
> We've experienced at least one version upgrade where they changed the database format and didn't provide any upgrade tools, so people were forced to start their metrics history from scratch. I know that it's pre-1.0, but still, they knew perfectly well that people were running it in production.
We broke backwards compatibility in the storage format once, and there's no plans to do so again. The core developers who were all running it in production didn't see it as worthwhile to write a converter, and noone else stepped up.
> The alert manager was also written from scratch recently, with a whole new config format.
The old alertmanager has always been flagged as very experimental, as it was a functioning PoC. The rewrite was always been on the cards, and this came up regularly.
This is all part of evolving the system to be better for everyone. If we tried to keep perfect backwards compatibility then we couldn't remove warts, bugs and misfeatures. We aren't afraid to deprecate where it makes sense to do so, and have transition plans where practical.
> The lack of packages (Debian/Ubuntu in our case) is also problematic.
There are packages in Debian proper, and nightlies at http://deb.robustperception.io/
> No syslog support is not acceptable in this day and age.
That's in the latest versions.
The high level problem is that there's so many different ways to do logging that we can't sanely support them all. For every X there is someone who thinks it's essential.
Evaluated it, and ultimately rejected it for a couple of reasons: - You must pick up Clojure to understand and configure Riemann (we're not a Clojure shop, so this is a non-trivial requirement) - Config file isn't a config file, it's an executed bit of Clojure code - Riemann is not a replacement for an alerting mechanism, it's another signal for alerting mechanisms (though since it's Clojure and the configuration fil…
> Config file isn't a config file, it's an executed bit of Clojure code For stream processing engines, configuration will be code. Unfortunate, but unavoidable. > Riemann is not a replacement for an alerting mechanism > Riemann is not a replacement for a trend graphing mechanism. Indeed it is not. It's misadvertised as a monitoring solution, while it's a stream processing engine. What I think of it is that you're sup…
How so?
Kafka is a stream processing engine that uses plain old Zookeeper data structures for config.
Edit: Kafka also seems to have the missing features you mentioned if Riemann should be taken seriously as a general-purpose stream processing engine.
Earlier quoted context omitted.
> Config file isn't a config file, it's an executed bit of Clojure code For stream processing engines, configuration will be code. Unfortunate, but unavoidable. > Riemann is not a replacement for an alerting mechanism > Riemann is not a replacement for a trend graphing mechanism. Indeed it is not. It's misadvertised as a monitoring solution, while it's a stream processing engine. What I think of it is that you're sup…
That was my problem with Riemann. I love its core but I really want something built on top of it. Basically a Jenkins of monitoring (since clojure is JVM). I contemplated building it (ie taking Jenkins plugin system as inspiration) but it was just way to much work.
Earlier quoted context omitted.
We're using Prometheus. Very happy, though the collection side of things aren't as smooth as we'd like it. The lack of a plugin system, and the reliance on HTTP to collect, means writing small collectors is a pain. You can't run 23 different daemons, each on their own port, to collect stats from things like PostgreSQL stats or RabbitMQ. We opted for the "text directory" way, where we populate a directory with .prom f…
> The lack of a plugin system We have many ways to plugin to Prometheus across the ecosystem, the textfile collector you're using is one of them. > You can't run 23 different daemons, each on their own port, to collect stats from things like PostgreSQL stats or RabbitMQ. There's no fundamental challenge with this approach. If you've got good basic infrastructure, particularly configuration management, the rollout of…
We do have good basic infrastructure, thanks. We use Puppet and have a decent deploy system that performs atomic deploys from Git.
We also do think in terms of services. But the exporter has to run somewhere. About half of our exporters are machine-specific (reads local stats from files or proc or whatever), about half run on the Prometheus node itself and talk to services like ElasticSearch or Postgres.
The problem is operational overhead of maintaining a dozen daemons per box, each with its own allocated port. It's not rocket science, just annoying. What could be a small script becomes something unnecessarily big. That's why we're sticking to "textfile" for now. When I have time, my plan is to write a small HTTP server that spawns plugin as subprocesses that emit their metrics via stdout, which seems like a much more reasonable, low-maintenance solution, and something node_exporter ought to support in the first place.