Introducing Heka • Hacker News Board

Re: Introducing Heka

#41

Post by lucian1900 » Wed, May 01, 2013, 10:06 AM UTC

tptacek wrote: post #34

coldtea wrote: post #32

Earlier quoted context omitted.

Quite easily. All offer options for building standalone programs that don't need a pre-installed runtime. You just copy them to some directory, run them and they work. And some of them even support building native binaries (e.g Java through gcc).

Virtually nobody in practice uses any of these.† Java binaries are in practice JVM bytecode in classfiles. Python programs are run by the Python interpreter. Go compiles to native code. Not only do you not need a preinstalled Go runtime on a target system, but there's very little advantage to even having one. The normal way of installing a Golang program is simply to copy the binary and run it. That's powerfully simp…

With the gigantic disadvantage of security updates requiring recompiling everything :(

Re: Introducing Heka

#42

Post by mh- » Wed, May 01, 2013, 10:19 AM UTC

aphyr wrote: post #38

tptacek wrote: post #34

Earlier quoted context omitted.

Virtually nobody in practice uses any of these.† Java binaries are in practice JVM bytecode in classfiles. Python programs are run by the Python interpreter. Go compiles to native code. Not only do you not need a preinstalled Go runtime on a target system, but there's very little advantage to even having one. The normal way of installing a Golang program is simply to copy the binary and run it. That's powerfully simp…

Huh. Most of the JVM shops I've worked at deploy apps as a monolithic fat .jar, built by their CI system, rather than trying to manage libraries on classpath.

I think GP comment was referring to the JVM 'runtime'. which, to me, is less of a complication than the Python scenario.

On that note, Python packaging/deployment/repeatability is still a disaster. If you have code with dependencies on compiled C extensions, there are few good ways to deal with this in prod.

In summary, I think a lot of us find the idea of monolithic binaries appealing (perhaps even to an irrational degree, speaking for myself) because of issues suffered in the past. :)

Re: Introducing Heka

#43

Post by mh- » Wed, May 01, 2013, 10:21 AM UTC

kevinmeredith wrote: post #4

Is this similar to piwik?

in that they're both on the internet.

i believe that's about it, though.

Re: Introducing Heka

#44

Post by Sven7 » Wed, May 01, 2013, 1:32 PM UTC

berkay wrote: post #29

Having implemented similar solutions, it's clear to me that developers did their homework and designed accordingly. I find myself agreeing with almost every decision I could see. - Go: light, no dependencies. This is key. If you ever deployed something in a non homogeneous environment with 100s/1000s of servers, you'd know the pain. - Plugin system: Only way to scale the development of the solution - Lua for plugins:…

Not very familiar with Go, could someone please elaborate on what "Go: no dependencies" mean? Thanks

Re: Introducing Heka

#45

Post by doun » Wed, May 01, 2013, 2:28 PM UTC

seems that this DO NOT support Windows?

Re: Introducing Heka

#46

Post by berkay » Wed, May 01, 2013, 2:52 PM UTC

Sven7 wrote: post #44

berkay wrote: post #29

Having implemented similar solutions, it's clear to me that developers did their homework and designed accordingly. I find myself agreeing with almost every decision I could see. - Go: light, no dependencies. This is key. If you ever deployed something in a non homogeneous environment with 100s/1000s of servers, you'd know the pain. - Plugin system: Only way to scale the development of the solution - Lua for plugins:…

Not very familiar with Go, could someone please elaborate on what "Go: no dependencies" mean? Thanks

Basically it means that you can have statically linked executables that do not have dependencies to other libraries, and can be deployed by simply copying the files.

Alternatives written is scripting language like ruby, python, etc. require runtimes and libraries, and it can get quite complicated to deploy them (especially if the env. has lots of different OS versions, etc.), keep track of all dependencies, deal with conflicts with other apps that may require a different version of the runtimes & libraries. As such for operational reasons it's very appealing to have a distributable binary that works without having to worry about what prerequisites are and whether it would impact anything else on the server.

Re: Introducing Heka

#47

Post by berkay » Wed, May 01, 2013, 3:04 PM UTC

snaky wrote: post #27

nonsequitarian wrote: post #26

Earlier quoted context omitted.

It's certainly not available out-of-box, and it would take a bit of work, but it should be possible to build something like this with Heka. You could write a dynamic filter plugin that watched for peaks in a specific graph (or any other arbitrary trigger) and, if found, generate a message to trigger the activation of a bunch of new dynamic filters, quickly turning on a more comprehensive set of data analyzers.

It's definitely possible, at least with RRDtool and 30 lines of Python (only 20 lines for react to measured peak, not forecasting one).

The question is whether reacting to peaks identified this way can be helpful or not. If you have hundreds of thousands of metrics, how many peaks get detected per minute and how many of those indicate something that actually requires attention?

Re: Introducing Heka

#48

Post by trink » Wed, May 01, 2013, 3:46 PM UTC

doun wrote: post #45

seems that this DO NOT support Windows?

Windows support will be added. https://github.com/mozilla-services/heka/issues/145

Re: Introducing Heka

#49

Post by mapleoin » Wed, May 01, 2013, 3:52 PM UTC

jonastryggvi wrote: post #40

nonsequitarian wrote: post #20

Earlier quoted context omitted.

Another member of the Heka team here. Yes, there are a lot of options for managing Python deployments. But none of Python's stories are as nice as "Here's a single binary, put this on every machine."

Exactly! I started playing with Go a few days ago, and immediately started thinking that this would be the perfect language to create something like LogStash, Flume or SplunkAgents in - and install to the machines that need to forward data to our centralised logging system. It really bugs me that I have to have an Python interpeter on the frontend web machines (cause I would prefer not to have a C compiler there)..

> It really bugs me that I have to have an Python interpeter on the frontend web machines (cause I would prefer not to have a C compiler there)..

You don't need to have a C compiler installed for the Python interpreter to work. I hope you're joking...

Re: Introducing Heka

#50

Post by marshray » Wed, May 01, 2013, 4:30 PM UTC

lucian1900 wrote: post #41

tptacek wrote: post #34

Earlier quoted context omitted.

Virtually nobody in practice uses any of these.† Java binaries are in practice JVM bytecode in classfiles. Python programs are run by the Python interpreter. Go compiles to native code. Not only do you not need a preinstalled Go runtime on a target system, but there's very little advantage to even having one. The normal way of installing a Golang program is simply to copy the binary and run it. That's powerfully simp…

With the gigantic disadvantage of security updates requiring recompiling everything :(

I've never seen an organization that didn't do a full rebuild of every build product contained in each release anyway. Usually it's just faster and less error-prone to do a full rebuild than to recompile the minimal set of source files and relink.