Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Grafana 4.0 with alerting is released

grafana.org

Post reply on HN
Print view View original

61–70 of 91 posts

Re: Grafana 4.0 with alerting is released

#61

Post by cheald »

mattkrea wrote: post #36
cheald wrote: post #16

Influx + Telegraf + Grafana is such a simple, sweet stack. No work to maintain, trivial to set up, I can ship just about anything I want into it, and reporting is fast. With alerting in place now, I'm even happier than ever. A huge thank you to the Grafana team for solving a huge pain point!

What kind of volume are you sending into Influx? It crashed on me probably 5 times a day with only 100 requests per second.

Right now it looks like it's around 50/sec. A lot of data points get rolled up by Telegraf on individual machines, and then it's shipped in via the UDP line protocol. I've written much larger volumes, though, and never had an issue with stability.

Re: Grafana 4.0 with alerting is released

#62

Post by cheald »

RRRA wrote: post #50
alfalfasprout wrote: post #25

Earlier quoted context omitted.

Not sure what you mean "secure telegraph into influxdb" but we've had great success with this stack for monitoring by just embedding an HTTP server into each application that needs to be monitored. We keep the HTTP server separate from any others used by the application (i.e. it runs on a separate thread) so performance isn't impacted.

My use case is one where I have servers in different datacenters and would want to have a simple, but secure, way to fetch metrics for graphing and alerts. So, I meant encryption in transport, authentication, etc. as many solutions work well if you're monitoring "in the clear" from the backend, but not so much over the internet.

We're deployed on AWS in multiple regions with VPNs set up between VPCs. No particular attention paid to securing the transport between Telegraf and Influx at the moment since a) it's either in an internal VPC or secured via ipsec, and b) our monitoring data is low-value enough that it doesn't warrant its own secure transport.

Re: Grafana 4.0 with alerting is released

#63

Post by jmedefind »

user5994461 wrote: post #35
jmedefind wrote: post #32

Earlier quoted context omitted.

I don't see how anyone can afford SaaS metrics/alert services at any sort of real scale. $15/month/host gets expensive fast. Datadog doesn't start providing discounts till you are at 1000+ hosts.

All vendors provide discount if you negotiate. ;) $15 * 500 hosts = $7500 per month. If you think it's expensive, I can only advise you to check how much the hardware will costs on EC2 to run the free tools, plus how much work it will take to get the 8 different and independent OSS tools to work not only alone but integrate together, plus how much additional work and maintenance to keep it working without hiccups (wa…

Oh I agree. That's why we ditched ec2 for our own bare metal cloud based on joyent and saved over $200k/year

Re: Grafana 4.0 with alerting is released

#64

Post by user5994461 »

neves wrote: post #53
user5994461 wrote: post #30

Quick note for the ones who are tired of the giant clusterfuck of open-source tools for monitoring + alerting + storage + other, which is no less than: - statsd - collectd - graphite - whisper - carbon - prometheus - grafana - seyren - riemann - nagios - icinga - zabbix There are multiple modern SaaS software that will do all of that in a single tool with better integrations, more polish, less work and no maintenance…

And you have include in the price the problems of a paid app: - customization will be very expensive, if not impossible - you must have people for the procurement process (x10 more costly if you are in a gov agency), - weird failures due to not finding the license, - your cheap personal that install software won't be able to do it, - you'll have problems creating testing environments because you don't have licenses -…

> - customization will be very expensive, if not impossible

You've got a full API and integrations with a hundred different tools and services out of the box.

Seriously, my coworker was skeptic at first too (so was I). Then we configured the full integrations with AWS/the-agent/statsd/postgre/mysql/cassandra/elasticsearch/riak/nginx/haproxy/redis/memcache/pagerduty/slack and some more.

My co-worker concluded in front of my CEO, "it was 2 orders of magnitude faster [than anything else we've ever tried for monitoring]". And that's not even talking about the additional features and customization we couldn't even dream of.

> - you must have people for the procurement process (x10 more costly if you are in a gov agency),

True. That's the only major problem I can see: People who can't buy the software they need. That's a social problem, not a software problem.

> - weird failures due to not finding the license

It's only one API key to put in the agent config file.

> your cheap personal that install software won't be able to do it

I don't know who you're talking about. Monitoring has our best people working on it. At other places I've seen, it's done by devops consultants raking up £600 a day.

There is no cheap personal involved. (Maybe you're thinking about of cheap interns who add alerts? that's an anti pattern).

- you'll have problems creating testing environments because you don't have licenses

Same license. Put a tag environment= in the config and done, all metrics all servers and all alerts will be tagged.

- you won't be able to do some things immediately because there aren't enough licenses.

Not applicable. It's not a limited license by seats.

You pay the bill at the end of the month depending on the number of hosts in your package. There is a hourly price for ephemeral hosts and overrun.

Re: Grafana 4.0 with alerting is released

#65

Post by user5994461 »

poezn wrote: post #33

Is anyone using a log management tool in conjunction with Grafana? I.e. if you see something anomalous or see an alert triggered, how do you investigate what's going on?

You can use Graylog for log management, that's the free open-source solution. (graylog + elasticsearch + mongodb)

You can use Splunk if you have money. That's the de facto standard. Beware that it's one of the most expensive software license on the planet :D

Re: Grafana 4.0 with alerting is released

#66

Post by user5994461 »

cheald wrote: post #61
mattkrea wrote: post #36

Earlier quoted context omitted.

What kind of volume are you sending into Influx? It crashed on me probably 5 times a day with only 100 requests per second.

Right now it looks like it's around 50/sec. A lot of data points get rolled up by Telegraf on individual machines, and then it's shipped in via the UDP line protocol. I've written much larger volumes, though, and never had an issue with stability.

If I may ask. How is UDP doing for you?

I checked my graphite setup once. We had 27% of metrics lost over UDP. That was bad.

pro-tip: "netstat -anus" and look at the error counters.

Re: Grafana 4.0 with alerting is released

#67

Post by jrv »

all_usernames wrote: post #31
user5994461 wrote: post #30

Quick note for the ones who are tired of the giant clusterfuck of open-source tools for monitoring + alerting + storage + other, which is no less than: - statsd - collectd - graphite - whisper - carbon - prometheus - grafana - seyren - riemann - nagios - icinga - zabbix There are multiple modern SaaS software that will do all of that in a single tool with better integrations, more polish, less work and no maintenance…

For installations of a few hundred instances or more, some of the SaaS offerings cost more than the engineering salaries it would take to maintain the OSS tools.

Case in point: http://blog.runnable.com/post/153498635761/how-we-saved-98-o...

Re: Grafana 4.0 with alerting is released

#68

Post by jrv »

user5994461 wrote: post #26
pfranz wrote: post #11

I'm currently using Prometheus, Grafana, and Alertmanager. I'm a big fan of the linux terminal, versioned config files, and separation of concerns but the rest of my team prefers web interfaces so I'm basically the only one maintaining Alertmanager. Grafana Altering looks appealing. What have other people had success with?

I've had success with killing all the s * free open source tools (Grafana, graphite, prometheus, whisper, icinga, nagios, carbon, ganglia, influxdb, zabbix...) And using a single paid tool that does the job better AND doesn't kill me in maintenance work. See https://www.datadoghq.com/ as leader or https://signalfx.com/ as the second comer, or http://www.bmcsoftware.uk/it-solutions/truesight.html if you're enterprisey…

See http://blog.runnable.com/post/153498635761/how-we-saved-98-o... as a counter point.

Re: Grafana 4.0 with alerting is released

#69

Post by jrv »

user5994461 wrote: post #30

Quick note for the ones who are tired of the giant clusterfuck of open-source tools for monitoring + alerting + storage + other, which is no less than: - statsd - collectd - graphite - whisper - carbon - prometheus - grafana - seyren - riemann - nagios - icinga - zabbix There are multiple modern SaaS software that will do all of that in a single tool with better integrations, more polish, less work and no maintenance…

Is there any hosted monitoring solution that integrates with service discovery, so that it's actually useful for serious alerting in nowadays' dynamic environments? Otherwise you can't even tell if things that should be there are reporting in or missing.

Re: Grafana 4.0 with alerting is released

#70

Post by dmitrios »

nopzor wrote: post #45
user5994461 wrote: post #30

Quick note for the ones who are tired of the giant clusterfuck of open-source tools for monitoring + alerting + storage + other, which is no less than: - statsd - collectd - graphite - whisper - carbon - prometheus - grafana - seyren - riemann - nagios - icinga - zabbix There are multiple modern SaaS software that will do all of that in a single tool with better integrations, more polish, less work and no maintenance…

Agree that the SaaS offerings are a lot more turnkey; the integrations and polish that make all the difference. What you call a 'clusterfuck' is really a wider ecosystem. It would be pretty crazy for a single organization to use all or even most of the tools that you list. Right now, people accept high degrees of cost (especially for at scale users) and lock-in, in exchange for the convenience of SaaS. Or, they go op…

As long as enterprises can will understand that they can get support options for Grafana(on-prem, SaaS, etc.) it just comes down to choosing the most economic option. I see benefit in symmetry for enterprise who has hybrid or still mostly in their datacenter.
Post reply on HN