Live data from Hacker News

Introducing Heka

blog.mozilla.org

51–58 of 58 posts

Re: Introducing Heka

#51

Earlier quoted context omitted.

With the gigantic disadvantage of security updates requiring recompiling everything :(

I've never seen an organization that didn't do a full rebuild of every build product contained in each release anyway. Usually it's just faster and less error-prone to do a full rebuild than to recompile the minimal set of source files and relink.

If one uses dynamic linking, one can use (some) system-provided libraries, which will get security updates in the usual manner.

Re: Introducing Heka

#52

Earlier quoted context omitted.

I've never seen an organization that didn't do a full rebuild of every build product contained in each release anyway. Usually it's just faster and less error-prone to do a full rebuild than to recompile the minimal set of source files and relink.

If one uses dynamic linking, one can use (some) system-provided libraries, which will get security updates in the usual manner.

It looks like Go supports dynamic linking to "system" libraries. At least on MS Windows this https://code.google.com/p/go/codesearch#go/src/cmd/dist/wind... call to FormatMessageW http://msdn.microsoft.com/en-us/library/windows/desktop/ms67... would be to an implementation in Kernel32.dll that would receive security updates.

On Linux there's a large gray area for things like libexpat.so.1 that may or may not be linked dynamically. But libc is LGPL, so I expect it too would be linked dynamically.

Re: Introducing Heka

#54
post #39
post #2

Seems similar to Riemann: http://riemann.io/

Riemann is top of our list to be implemented in an environment made up of PostgreSQL, Memcached, Python (Django) and Go. But as Hekad has plugins for all of the above (except Go - but I'm sure it's possible): http://heka-docs.readthedocs.org/en/latest/architecture/inde... Well I guess we'll now be evaluating whether Hekad looks like it might be a more promising fit. I particularly like the bullet points on aggregatio…

Unfortunately, the diagram you linked to is misleading; we don't have all of those plugins built yet. We've got a (quite rich) Python client (https://github.com/mozilla-services/heka-py) and a (rudimentary) Go client (https://github.com/mozilla-services/heka/tree/master/client) but Memcached and PostgreSQL connectors aren't yet in place. Fleshing out our plugin set (especially the inputs) is one of our highest priorities, so those should be coming Real Soon Now(tm). Contributions welcome!

Re: Introducing Heka

#55
post #24

Uh, yet another collector/grapher. That's nice but.. We have tons of collectors. And tons of graphers. What we have not is a little bit of smarts in that tools. Ability to predict and ability to react. Predict. We have Holt-Winters Forecasting Algorithm implemented in RRDTool from 2005 and a couple of papers. React. I'm not talking about 'fix it automagically'. But everyone wants to know 'wtf was that peak on this gr…

We're working on it here at Etsy :) It'll be released in a week or two. In the meantime, I've been speaking about it: http://devslovebacon.com/conferences/bacon-2013/talks/bring-...

That's interesting. Can I subscribe to announce?

Re: Introducing Heka

#56
post #47
post #27

Earlier quoted context omitted.

It's definitely possible, at least with RRDtool and 30 lines of Python (only 20 lines for react to measured peak, not forecasting one).

The question is whether reacting to peaks identified this way can be helpful or not. If you have hundreds of thousands of metrics, how many peaks get detected per minute and how many of those indicate something that actually requires attention?

There are many ways to sort peaks out. For instance, there is a script for RRDtool that removes obviously irrelevant spikes [1].

The vast majority of hundreds of thousands metrics are common for any node/server actually, so predefined recipes/settings should be used for them. Only aplication-level metrics for only in-house applications have to be tuned manually.

[1] http://oss.oetiker.ch/rrdtool/pub/contrib/removespikes-20080...

Re: Introducing Heka

#57
post #34

Earlier quoted context omitted.

Virtually nobody in practice uses any of these.† Java binaries are in practice JVM bytecode in classfiles. Python programs are run by the Python interpreter. Go compiles to native code. Not only do you not need a preinstalled Go runtime on a target system, but there's very little advantage to even having one. The normal way of installing a Golang program is simply to copy the binary and run it. That's powerfully simp…

With the gigantic disadvantage of security updates requiring recompiling everything :(

Yeah, true, each time that happens, it'll be 450ms of your life you'll never get back.

Re: Introducing Heka

#58
post #24

Uh, yet another collector/grapher. That's nice but.. We have tons of collectors. And tons of graphers. What we have not is a little bit of smarts in that tools. Ability to predict and ability to react. Predict. We have Holt-Winters Forecasting Algorithm implemented in RRDTool from 2005 and a couple of papers. React. I'm not talking about 'fix it automagically'. But everyone wants to know 'wtf was that peak on this gr…

We're working on algorithms to improve the 'predict' part (way beyond Holt-Winters!): ow.ly/jRKrT

If you're interested to see our algorithms in action, email me jennyinc at gmail.com

Post reply on HN