Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Reaching 200K events/sec

aphyr.com

Post reply on HN
Print view View original

21–29 of 29 posts

Re: Reaching 200K events/sec

#21

Post by aphyr »

revelation wrote: post #4

TLDR: Burned by framework magic. Talk about side effects. Ten layers (and probably buffers) traveled through until your data hits the wire. Layer x decides to change its IO model and your throughput takes a dive. It's exactly why there was a post recently about building an operating system just to run some network daemon.

I dunno if I'd call it magic, per se. Netty is a big library and it has good reasons for behaving this way. I just don't think I came through the right documentation path is all.

Re: Reaching 200K events/sec

#22

Post by dschiptsov » 

  (defn execution-handler
    "Creates a new netty execution handler."
    []
    (ExecutionHandler.
      (OrderedMemoryAwareThreadPoolExecutor.
        16       ; Core pool size
        1048576  ; 1MB per channel queued
        10485760 ; 10MB total queued
        )))
It is a farce, isn't it?))

Re: Reaching 200K events/sec

#23

Post by aphyr »

dschiptsov wrote: post #22

(defn execution-handler "Creates a new netty execution handler." [] (ExecutionHandler. (OrderedMemoryAwareThreadPoolExecutor. 16 ; Core pool size 1048576 ; 1MB per channel queued 10485760 ; 10MB total queued ))) It is a farce, isn't it?))

Heh, I originally cargo-culted this line from somewhere in Netty's docs: https://github.com/aphyr/riemann/blob/732a6e8986f75c638c30b5...

After finally digging into how execution handlers worked, the puzzle started to unravel. Netty's docs are pretty good, but you have to understand what all the names mean before you can understand, well, any one part of the system. Bit tough to piece together, at least for my little brain. ;-)

Re: Reaching 200K events/sec

#24

Post by no7hing »

aphyr wrote: post #18
spullara wrote: post #13

Earlier quoted context omitted.

You are likely doing something wrong. My in-memory redis server clone that also uses netty reaches 1.5M requests/s on an arguably heavier workload with a more complicated protocol. I'd love to help you optimize it if it matters.

Thanks for the offer! I, uh, certainly didn't mean to claim that Riemann is extraordinarily fast. Up until this point I've been working as hard as I can just to make the darn thing turn on and accomplish something useful. This is quite early in the optimization process, haha. :) The principle bottleneck in this particular test is actually the client--Riemann itself spends ~94% of its time waiting on epoll in this tes…

Things will probably look better once you switch your client implementation to Netty as a similar test I did with Netty 4, Protobufs and batching of messages produced low seven figure numbers too.

Re: Reaching 200K events/sec

#25

Post by dschiptsov »

aphyr wrote: post #23
dschiptsov wrote: post #22

(defn execution-handler "Creates a new netty execution handler." [] (ExecutionHandler. (OrderedMemoryAwareThreadPoolExecutor. 16 ; Core pool size 1048576 ; 1MB per channel queued 10485760 ; 10MB total queued ))) It is a farce, isn't it?))

Heh, I originally cargo-culted this line from somewhere in Netty's docs: https://github.com/aphyr/riemann/blob/732a6e8986f75c638c30b5... After finally digging into how execution handlers worked, the puzzle started to unravel. Netty's docs are pretty good, but you have to understand what all the names mean before you can understand, well, any one part of the system. Bit tough to piece together, at least for my little…

All I'm trying to say is that switching to the prefix notation adding parenthesis does not make Java a Lisp. ;-)

I'm also not sure that JVM itself is a good idea, especially for serving content, but I do respect people who are trying nevertheless.)

Re: Reaching 200K events/sec

#26

Post by thebluesky »

spullara wrote: post #13
aphyr wrote: post #5

If you're wondering about the workload, this is the trivial Riemann config this benchmark uses: (streams (rate 5 (comp prn float :metric)) Which means for each five-second interval, sum all the metrics of the events flowing through this stream, divide by the time elapsed, and print that rate to the console. I'm using this setup to put the heaviest load possible on Riemann's client and TCP server while I optimize thos…

You are likely doing something wrong. My in-memory redis server clone that also uses netty reaches 1.5M requests/s on an arguably heavier workload with a more complicated protocol. I'd love to help you optimize it if it matters.

Is the source available anywhere?

1.5M locally or over a network? What is the size of each message?

Re: Reaching 200K events/sec

#27

Post by spullara »

thebluesky wrote: post #26
spullara wrote: post #13

Earlier quoted context omitted.

You are likely doing something wrong. My in-memory redis server clone that also uses netty reaches 1.5M requests/s on an arguably heavier workload with a more complicated protocol. I'd love to help you optimize it if it matters.

Is the source available anywhere? 1.5M locally or over a network? What is the size of each message?

http://github.com/spullara/redis-protocol — server module

The test was run with "SET key value" which in the redis protocol is something like 15-20 bytes / message.

Re: Reaching 200K events/sec

#28

Post by spullara »

aphyr wrote: post #18
spullara wrote: post #13

Earlier quoted context omitted.

You are likely doing something wrong. My in-memory redis server clone that also uses netty reaches 1.5M requests/s on an arguably heavier workload with a more complicated protocol. I'd love to help you optimize it if it matters.

Thanks for the offer! I, uh, certainly didn't mean to claim that Riemann is extraordinarily fast. Up until this point I've been working as hard as I can just to make the darn thing turn on and accomplish something useful. This is quite early in the optimization process, haha. :) The principle bottleneck in this particular test is actually the client--Riemann itself spends ~94% of its time waiting on epoll in this tes…

Ah ok, my test was run using the C redis-benchmark client.

Re: Reaching 200K events/sec

#29

Post by aphyr »

dschiptsov wrote: post #25
aphyr wrote: post #23

Earlier quoted context omitted.

Heh, I originally cargo-culted this line from somewhere in Netty's docs: https://github.com/aphyr/riemann/blob/732a6e8986f75c638c30b5... After finally digging into how execution handlers worked, the puzzle started to unravel. Netty's docs are pretty good, but you have to understand what all the names mean before you can understand, well, any one part of the system. Bit tough to piece together, at least for my little…

All I'm trying to say is that switching to the prefix notation adding parenthesis does not make Java a Lisp. ;-) I'm also not sure that JVM itself is a good idea, especially for serving content, but I do respect people who are trying nevertheless.)

Some of Riemann's code is very much like Java, especially where I interoperate with Java libraries. Other parts of the code are... definitely not Java:

  (defn where-partition-clauses
    "Given expressions like (a (else b) c (else d)), returns [[a c] [b d]]"
    [exprs]
    (map vec
         ((juxt remove
                (comp (partial mapcat rest) filter))
            (fn [expr]
              (when (list? expr)
                (= 'else (first expr))))
            exprs)))
I wouldn't categorize Clojure as a "pure lisp"--it relies heavily on the JVM type system, for starters--but idiomatic Clojure feels closer to Lisp than Java, to me.
Post reply on HN