Live data from Hacker News

Introducing Heka

blog.mozilla.org

21–30 of 58 posts

Re: Introducing Heka

#21
post #19
post #9

Earlier quoted context omitted.

That doesn't make a lot of sense. You don't have to write a monitoring software from scratch just because you want statically compiled bundled libraries. You can do that with any programming language.

How do you run Python, Java, Perl, Ruby, or any JVM language without an installed runtime?

Technically, you could use something like PyInstaller[1] to bundle the runtime and all libraries into an executable package. Practically, no one ever does that. :)

[1] http://www.pyinstaller.org/

Re: Introducing Heka

#23
post #3

Very interesting. Does this fit in conceptually with circus at all? It seems like there's a fair amount of overlap between the process/HTTP management done by circus and this stats/data collection/analysis (specifically hekad agent in the architecture diagram): http://heka-docs.readthedocs.org/en/latest/architecture/inde... I'm curious if Mozilla is using these two tools in combination internally, and what that archi…

Heka and Circus have very different goals. Heka is about generic data gathering, processing, and routing. Circus is about task and process monitoring and management. There's some overlap in that Circus needs to gather and process a bit of data to do its job, and it would certainly be possible to use Heka as a part of that, but we're not doing so. We'll probably use Circus to manage at least a few of our `hekad` processes, though.

Re: Introducing Heka

#24
Uh, yet another collector/grapher. That's nice but..

We have tons of collectors. And tons of graphers. What we have not is a little bit of smarts in that tools. Ability to predict and ability to react.

Predict. We have Holt-Winters Forecasting Algorithm implemented in RRDTool from 2005 and a couple of papers.

React. I'm not talking about 'fix it automagically'. But everyone wants to know 'wtf was that peak on this graph last night?'. Usually your never know, except the simplest cases. Because you cannot collect everything about everything all the time. But monitoring system could enable 'collect everything we can' for short period of time when it detects something. Something wrong or something strange, something out of the pattern. Does anybody hear about system with something like that?

Re: Introducing Heka

#25
post #11

i see it more as a syslog replacement. It does a lot more than syslog of course, but tit doesnt do what "collectd" and whatever else does. Heka seems to "just" do logging/routing/etc and be extremely fast and reliable doing so. And has no dependencies/small footprint. Which is what syslog can't do.

Actually, there is a fair amount of overlap between collectd and Heka. And Heka provides mechanisms for in-flight data processing and graphing output, in addition to logging and routing. But you're also right that Heka might in some cases also be used in place of syslog.

Of course, both syslog and collectd have been around and battle-hardened for many(!) years, whereas this first Heka release is being called "0.2-beta-1" for a reason. I wouldn't go rush into replacing any mission-critical infrastructure just yet. ;)

Re: Introducing Heka

#26
post #24

Uh, yet another collector/grapher. That's nice but.. We have tons of collectors. And tons of graphers. What we have not is a little bit of smarts in that tools. Ability to predict and ability to react. Predict. We have Holt-Winters Forecasting Algorithm implemented in RRDTool from 2005 and a couple of papers. React. I'm not talking about 'fix it automagically'. But everyone wants to know 'wtf was that peak on this gr…

It's certainly not available out-of-box, and it would take a bit of work, but it should be possible to build something like this with Heka. You could write a dynamic filter plugin that watched for peaks in a specific graph (or any other arbitrary trigger) and, if found, generate a message to trigger the activation of a bunch of new dynamic filters, quickly turning on a more comprehensive set of data analyzers.

Re: Introducing Heka

#27
post #24

Uh, yet another collector/grapher. That's nice but.. We have tons of collectors. And tons of graphers. What we have not is a little bit of smarts in that tools. Ability to predict and ability to react. Predict. We have Holt-Winters Forecasting Algorithm implemented in RRDTool from 2005 and a couple of papers. React. I'm not talking about 'fix it automagically'. But everyone wants to know 'wtf was that peak on this gr…

It's certainly not available out-of-box, and it would take a bit of work, but it should be possible to build something like this with Heka. You could write a dynamic filter plugin that watched for peaks in a specific graph (or any other arbitrary trigger) and, if found, generate a message to trigger the activation of a bunch of new dynamic filters, quickly turning on a more comprehensive set of data analyzers.

It's definitely possible, at least with RRDtool and 30 lines of Python (only 20 lines for react to measured peak, not forecasting one).

Re: Introducing Heka

#28
post #5

Off the top of my head this is a reimplementation of the following * SNMP * CollectD * Carbon * JMX * WMI * CMIP And a whole host of other proprietary transports. So its cool and looks awesome, but what does it give me that the entirety of other monitoring protocols doesn't

SNMP -> shivers

Re: Introducing Heka

#29
Having implemented similar solutions, it's clear to me that developers did their homework and designed accordingly. I find myself agreeing with almost every decision I could see. - Go: light, no dependencies. This is key. If you ever deployed something in a non homogeneous environment with 100s/1000s of servers, you'd know the pain.

- Plugin system: Only way to scale the development of the solution

- Lua for plugins: Yes! Language is not important, but not having to stop and restart the application for changes in logic, etc. is essential.

- Routing. Sounds great, can't wait to take a deeper look.

Kudos to devs. Nicely done!

Post reply on HN