Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Reaching 200K events/sec

aphyr.com

Post reply on HN
Print view View original

11–20 of 29 posts

Re: Reaching 200K events/sec

#11

Post by aphyr »

trekkin wrote: post #6
aphyr wrote: post #3

Earlier quoted context omitted.

Ah, I didn't intend for this post to reach Hacker News absent context. If you haven't been tracking Riemann, this post might not make sense. ;-) Riemann is not an HTTP server, or anything analogous. It's an event processor, and reacts to incoming events by running them through an arbitrary set of functions. Events are the logical "requests" against the system, if you're thinking in HTTP terms. Messages are just a bun…

Well, I'm sure there are specific use cases where Riemann would be preferable to a generic web server. But for most developers in most situations, it is a no brainer to choose an HTTP-based protocol with off-the-shelf HTTPD server over a 10x slower proprietary system.

Er, I don't mean to be contrarian, but Riemann is anything but proprietary. I'm an unemployed OSS developer. Every bit of code from clients to dashboard to server to integration tools to the website is open source: http://aphyr.github.com/riemann/. I'm not trying to sell anything. I'm just building a tool to solve a problem I faced in developing distributed systems.

Second... I guess I can reiterate. Riemann is not an HTTP server. It's an event-stream driven monitoring system. The protocol uses existing standards (e.g. protobufs), is simple to implement, and the community has written clients for many languages: http://riemann.io/clients.html.

As an aside, I do plan on adding an HTTP interface to Riemann, but HTTP processing (and using JSON for serialization) comes with certain unavoidable costs in bandwidth, memory and latency. It'll fill a complementary space to the existing TCP and UDP interfaces.

Re: Reaching 200K events/sec

#12

Post by trekkin »

aphyr wrote: post #11
trekkin wrote: post #6

Earlier quoted context omitted.

Well, I'm sure there are specific use cases where Riemann would be preferable to a generic web server. But for most developers in most situations, it is a no brainer to choose an HTTP-based protocol with off-the-shelf HTTPD server over a 10x slower proprietary system.

Er, I don't mean to be contrarian, but Riemann is anything but proprietary. I'm an unemployed OSS developer. Every bit of code from clients to dashboard to server to integration tools to the website is open source: http://aphyr.github.com/riemann/ . I'm not trying to sell anything. I'm just building a tool to solve a problem I faced in developing distributed systems. Second... I guess I can reiterate. Riemann is not…

Sorry, by "proprietary" I meant "custom". I admire people who have skills and dedication to built OSS, this was just a wrong word to use.

I completely agree that for specific uses Riemann is great. Your post was, though, about the performance/throughput, and so my comment was about the performance/throughput. Streaming messages/events over the network is an old problem, with well-known limitations, and this was what my comment was about.

Re: Reaching 200K events/sec

#13

Post by spullara »

aphyr wrote: post #5

If you're wondering about the workload, this is the trivial Riemann config this benchmark uses: (streams (rate 5 (comp prn float :metric)) Which means for each five-second interval, sum all the metrics of the events flowing through this stream, divide by the time elapsed, and print that rate to the console. I'm using this setup to put the heaviest load possible on Riemann's client and TCP server while I optimize thos…

You are likely doing something wrong. My in-memory redis server clone that also uses netty reaches 1.5M requests/s on an arguably heavier workload with a more complicated protocol. I'd love to help you optimize it if it matters.

Re: Reaching 200K events/sec

#14

Post by aba_sababa »

trekkin wrote: post #12
aphyr wrote: post #11

Earlier quoted context omitted.

Er, I don't mean to be contrarian, but Riemann is anything but proprietary. I'm an unemployed OSS developer. Every bit of code from clients to dashboard to server to integration tools to the website is open source: http://aphyr.github.com/riemann/ . I'm not trying to sell anything. I'm just building a tool to solve a problem I faced in developing distributed systems. Second... I guess I can reiterate. Riemann is not…

Sorry, by "proprietary" I meant "custom". I admire people who have skills and dedication to built OSS, this was just a wrong word to use. I completely agree that for specific uses Riemann is great. Your post was, though, about the performance/throughput, and so my comment was about the performance/throughput. Streaming messages/events over the network is an old problem, with well-known limitations, and this was what…

Please don't criticize what he's built before you even barely understand it.

Re: Reaching 200K events/sec

#15

Post by bkirkbri »

aphyr wrote: post #10
sgrove wrote: post #8

Earlier quoted context omitted.

What similarities between the two pieces do you see? What task would you replace Riemann with httpd? Genuinely curious, though slightly skeptical.

They're completely different kinds of software, at least as I understand them. Riemann is about pushing lots of little hashmaps (events) through a DAG of streaming functions; HTTP is about synchronous gets/puts/updates/deletes to a tree of resources. If you are trying to make sense of Riemann in HTTP terms, sending an event to Riemann might look like POST /streams, with a body containing a single JSON object. There's…

Don't let the afterhours haters -- sorry, hackers -- get you down. Riemann looks to be a fantastic piece of open-source software for scratching a particular itch.

Coincidentally, I was just working on a tracing system to dump data to Riemann while eating HTTP logs and/or handling live requests from browsers. It seems to be just what we need to aggregate, monitor and graph our trace data. Thanks!

Re: Reaching 200K events/sec

#16

Post by jeremyjh »

trekkin wrote: post #9
rbranson wrote: post #7

Earlier quoted context omitted.

Did you even read what aphyr wrote before posting this? It detailed that unlike HTTP, which in general supplies one request per message (GET, POST, etc), a Riemann message contains potentially hundreds of different requests that must be processed individually.

You can put as many "events" in the body of an HTTP POST request as you wish. What really matters in distributed messaging systems, from the performance point of view, is the number of distinct messages per second. And if a system designer wants to send a stream of "events" to another system to be acted upon, and if this designer cares about throughput (which is assumed here, given the title of the post), then this d…

Is that why HTTP is always used for high-throughput network services like MySQL, Memcached or Redis? Oh wait it isn't.

Re: Reaching 200K events/sec

#18

Post by aphyr »

spullara wrote: post #13
aphyr wrote: post #5

If you're wondering about the workload, this is the trivial Riemann config this benchmark uses: (streams (rate 5 (comp prn float :metric)) Which means for each five-second interval, sum all the metrics of the events flowing through this stream, divide by the time elapsed, and print that rate to the console. I'm using this setup to put the heaviest load possible on Riemann's client and TCP server while I optimize thos…

You are likely doing something wrong. My in-memory redis server clone that also uses netty reaches 1.5M requests/s on an arguably heavier workload with a more complicated protocol. I'd love to help you optimize it if it matters.

Thanks for the offer! I, uh, certainly didn't mean to claim that Riemann is extraordinarily fast. Up until this point I've been working as hard as I can just to make the darn thing turn on and accomplish something useful. This is quite early in the optimization process, haha. :)

The principle bottleneck in this particular test is actually the client--Riemann itself spends ~94% of its time waiting on epoll in this test. The client is a total hack using Java OIO, calling flush() on every message, Nagle's algorithm disabled for low per-msg latencies... it's a wreck, haha. Part of next week's optimization push is replacing it with a Netty client and tuning TCP options for various workloads.

Re: Reaching 200K events/sec

#19

Post by aphyr »

bkirkbri wrote: post #15
aphyr wrote: post #10

Earlier quoted context omitted.

They're completely different kinds of software, at least as I understand them. Riemann is about pushing lots of little hashmaps (events) through a DAG of streaming functions; HTTP is about synchronous gets/puts/updates/deletes to a tree of resources. If you are trying to make sense of Riemann in HTTP terms, sending an event to Riemann might look like POST /streams, with a body containing a single JSON object. There's…

Don't let the afterhours haters -- sorry, hackers -- get you down. Riemann looks to be a fantastic piece of open-source software for scratching a particular itch. Coincidentally, I was just working on a tracing system to dump data to Riemann while eating HTTP logs and/or handling live requests from browsers. It seems to be just what we need to aggregate, monitor and graph our trace data. Thanks!

Thank you. :) If you have any questions, feel free to hop on Freenode #riemann and I'll do my best to help out.

BTW, you're not the first to wonder about streaming events directly to Riemann from client browsers. I... don't recommend it, just because I don't have the time to appropriately guarantee Riemann's performance and security characteristics as an internet-facing service (yet), but adding an HTTP POST path to (ws-server) is definitely on my list. Even if the HTTP+JSON interface is much slower than the TCP/UDP interfaces, I think it'll be plenty useful for many deployments, especially those making requests from JS.

Re: Reaching 200K events/sec

#20

Post by askimto »

revelation wrote: post #4

TLDR: Burned by framework magic. Talk about side effects. Ten layers (and probably buffers) traveled through until your data hits the wire. Layer x decides to change its IO model and your throughput takes a dive. It's exactly why there was a post recently about building an operating system just to run some network daemon.

What post was that?
Post reply on HN