Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Introducing Heka

blog.mozilla.org

Post reply on HN
Print view View original

31–40 of 58 posts

Re: Introducing Heka

#31

Post by aba_sababa »

snaky wrote: post #24

Uh, yet another collector/grapher. That's nice but.. We have tons of collectors. And tons of graphers. What we have not is a little bit of smarts in that tools. Ability to predict and ability to react. Predict. We have Holt-Winters Forecasting Algorithm implemented in RRDTool from 2005 and a couple of papers. React. I'm not talking about 'fix it automagically'. But everyone wants to know 'wtf was that peak on this gr…

We're working on it here at Etsy :)

It'll be released in a week or two. In the meantime, I've been speaking about it: http://devslovebacon.com/conferences/bacon-2013/talks/bring-...

Re: Introducing Heka

#32

Post by coldtea »

tptacek wrote: post #19
mapleoin wrote: post #9

Earlier quoted context omitted.

That doesn't make a lot of sense. You don't have to write a monitoring software from scratch just because you want statically compiled bundled libraries. You can do that with any programming language.

How do you run Python, Java, Perl, Ruby, or any JVM language without an installed runtime?

Quite easily. All offer options for building standalone programs that don't need a pre-installed runtime.

You just copy them to some directory, run them and they work.

And some of them even support building native binaries (e.g Java through gcc).

Re: Introducing Heka

#33

Post by coldtea »

keyle wrote: post #22

Off topic but I did smile to see it's written in Go, and not Rust. I guess Rust isn't there yet.

What's to smile about?

Nobody ever claimed that Rust is "there yet".

The core Rust developers all say that Rust is still in flux, and that a stable version is still many months in the future, possibly 2014. And they advise not to use it in production.

Re: Introducing Heka

#34

Post by tptacek »

coldtea wrote: post #32
tptacek wrote: post #19

Earlier quoted context omitted.

How do you run Python, Java, Perl, Ruby, or any JVM language without an installed runtime?

Quite easily. All offer options for building standalone programs that don't need a pre-installed runtime. You just copy them to some directory, run them and they work. And some of them even support building native binaries (e.g Java through gcc).

Virtually nobody in practice uses any of these.† Java binaries are in practice JVM bytecode in classfiles. Python programs are run by the Python interpreter.

Go compiles to native code. Not only do you not need a preinstalled Go runtime on a target system, but there's very little advantage to even having one. The normal way of installing a Golang program is simply to copy the binary and run it. That's powerfully simpler than most other modern programming languages, with the obvious exception(s) of C/C++/ObjC.

† Commenter downthread says the same thing, but let me add that we look at other people's Python/Java/Ruby programs professionally, and I can't recall a single client ever doing anything like this.

Re: Introducing Heka

#35

Post by grosskur »

I've been experimenting with a different metrics toolchain of shh + log-shuttle + l2met recently (also written in Go):

https://github.com/freeformz/shh

https://github.com/ryandotsmith/log-shuttle

https://github.com/ryandotsmith/l2met

shh can be extended with custom pollers written in Go, but focuses on collecting system-level metrics. log-shuttle is a general-purpose tool for shipping logs over HTTP. l2met receives logs over HTTP and can be extended with custom outlets written in Go, but requires log statements in a specific format ("measure.db.latency=20" or "measure=db.latency val=20").

It's great to see so many new tools in this space. Previously I had a bunch of one-off "carbonize" scripts running out of cron, each collecting a specific kind of metric and sending it to Graphite or statsd. This worked OK but required quite a bit of code to get things done. Heka's plugin system looks like a nice way to structure things.

Re: Introducing Heka

#36

Post by coldtea »

tptacek wrote: post #34
coldtea wrote: post #32

Earlier quoted context omitted.

Quite easily. All offer options for building standalone programs that don't need a pre-installed runtime. You just copy them to some directory, run them and they work. And some of them even support building native binaries (e.g Java through gcc).

Virtually nobody in practice uses any of these.† Java binaries are in practice JVM bytecode in classfiles. Python programs are run by the Python interpreter. Go compiles to native code. Not only do you not need a preinstalled Go runtime on a target system, but there's very little advantage to even having one. The normal way of installing a Golang program is simply to copy the binary and run it. That's powerfully simp…

>Virtually nobody in practice uses any of these.† Java binaries are in practice JVM bytecode in classfiles. Python programs are run by the Python interpreter.

The "Virtually nobody" this is because the main use case for Python and Java are as server side languages (both) and scripting languages (Python). In those cases people are expected to have or to setup the appropriate runtime beforehand.

But for people who want to ship apps to end users (customers and consumers) with Java and Python, the bundling thing is very very common.

People using them in the end user space, regularly do it this exact way. For most of them, you don't even get to know what they use underneath.

Some examples:

- Dropbox (uses and bundles Python in the app).

- Vuze torrent client (previously Azureus and very popular in its prime) bundles a JRE (for when you don't have an installed one).

- LightTable is just a JS runtime bundled with Webkit as a standalone app.

Re: Introducing Heka

#37

Post by EFruit »

aba_sababa wrote: post #31
snaky wrote: post #24

Uh, yet another collector/grapher. That's nice but.. We have tons of collectors. And tons of graphers. What we have not is a little bit of smarts in that tools. Ability to predict and ability to react. Predict. We have Holt-Winters Forecasting Algorithm implemented in RRDTool from 2005 and a couple of papers. React. I'm not talking about 'fix it automagically'. But everyone wants to know 'wtf was that peak on this gr…

We're working on it here at Etsy :) It'll be released in a week or two. In the meantime, I've been speaking about it: http://devslovebacon.com/conferences/bacon-2013/talks/bring-...

Is what you're referring to something like Growl that pops up and says 'Hey, metrics.sessions.active just dropped by 70%", or something pre-configured to spin up additional VMs/instances/dynos when some metrics misbehave? TL;DR: How autonomous is it?

Re: Introducing Heka

#38

Post by aphyr »

tptacek wrote: post #34
coldtea wrote: post #32

Earlier quoted context omitted.

Quite easily. All offer options for building standalone programs that don't need a pre-installed runtime. You just copy them to some directory, run them and they work. And some of them even support building native binaries (e.g Java through gcc).

Virtually nobody in practice uses any of these.† Java binaries are in practice JVM bytecode in classfiles. Python programs are run by the Python interpreter. Go compiles to native code. Not only do you not need a preinstalled Go runtime on a target system, but there's very little advantage to even having one. The normal way of installing a Golang program is simply to copy the binary and run it. That's powerfully simp…

Huh. Most of the JVM shops I've worked at deploy apps as a monolithic fat .jar, built by their CI system, rather than trying to manage libraries on classpath.

Re: Introducing Heka

#39

Post by buro9 »

judofyr wrote: post #2

Seems similar to Riemann: http://riemann.io/

Riemann is top of our list to be implemented in an environment made up of PostgreSQL, Memcached, Python (Django) and Go.

But as Hekad has plugins for all of the above (except Go - but I'm sure it's possible): http://heka-docs.readthedocs.org/en/latest/architecture/inde...

Well I guess we'll now be evaluating whether Hekad looks like it might be a more promising fit.

I particularly like the bullet points on aggregation counters, filters and transformations. We'll have to see how they work in practise though. The docs are very pretty, but as is usual with early releases it seems a little difficult to picture the whole and how it will actually work in practise from the soup of detail that Sphinx spits out.

Re: Introducing Heka

#40

Post by jonastryggvi »

nonsequitarian wrote: post #20
mapleoin wrote: post #13

Earlier quoted context omitted.

> Statically linking the world isn't trivial. For our existing Python code bases - how are you going to deal with third party libraries from PyPI? Depends on what you want? You could freeze the pip-requires to always install the same version and use a virtualenv per application. This is basically the same as bundling everything together, it has all the benefits with the least amount of work. You could use distributio…

Another member of the Heka team here. Yes, there are a lot of options for managing Python deployments. But none of Python's stories are as nice as "Here's a single binary, put this on every machine."

Exactly! I started playing with Go a few days ago, and immediately started thinking that this would be the perfect language to create something like LogStash, Flume or SplunkAgents in - and install to the machines that need to forward data to our centralised logging system.

It really bugs me that I have to have an Python interpeter on the frontend web machines (cause I would prefer not to have a C compiler there)..

Post reply on HN