Storm - the Hadoop of realtime processing

Post by herdrick » Thu, May 26, 2011, 6:33 PM UTC

nathanmarz wrote: post #19

herdrick wrote: post #17

This sounds great. This is the traditional realtime processing use case: process messages and update a variety of databases. Question: I typically think of real-time as a need for user-facing things, i.e. handling a user's requests before he gets bored and goes away. Is Storm set up for that? Or is it mostly meant to update a database with results rather than return them to a waiting process?

It handles both cases. Storm can be used to asynchronously update databases in realtime in a scalable way (replacing traditional systems of queues and workers). Using Storm for Distributed RPC lets you do intense computations on Storm and return them to a waiting process.

Post by nathanmarz » Thu, May 26, 2011, 6:37 PM UTC

scott_s wrote: post #20

bmohlenhoff wrote: post #18

It sounds like a neat project, but I think describing it as "real time" is misleading if you're not also providing information on latency. The majority of the provided use cases seem to indicate a high level of scalability and durability, as well as a high level of throughput, but these are not necessary characteristics of a true real time system. It's a common misconception. A real time system doesn't have to be fas…

You described a hard real-time system. That exists for things like the controllers on a jet. What's becoming much more prevalent are soft real-time systems that perform analytics. There won't be any catastrophic failure if deadlines aren't meant - and there may not even be any expressed deadline - it's just understood that the data must be processed and analyzed as fast as possible to be useful.

Post by jbellis » Thu, May 26, 2011, 6:41 PM UTC

Maro wrote: post #14

I'm not sure if this is the same thing, but there's also a new company called Hadapt (the commercialization of HadoopDB). It's about adapting Hadoop for real-time analytic SQL queries by putting local SQL dbs on the Hadoop nodes and then using the Hadoop plumbing. It's based on Daniel Abadi's research, he's a really smart guy.

Post by herdrick » Thu, May 26, 2011, 6:45 PM UTC

nathanmarz wrote: post #11

maxdemarzi wrote: post #8

" To compute reach, you need to get all the people who tweeted the URL, get all the followers of all those people, unique that set of followers, and then count the number of uniques. It's an intense computation that potentially involves thousands of database calls and tens of millions of follower records." Or you could use a Graph DB to solve a Graph problem. URL -> tweeted_by -> users -> followed_by -> users Try tha…

To do that query on Neo4j, you would need to store in memory on one machine the entire Twitter social graph, all the people who tweeted every URL ever tweeted on Twitter, and then do the computation on a single thread. Neo4j can't handle that scale. The reach computation on Storm does everything in parallel (across however many machines you need to scale the computation) and gets data using distributed key/value data…

Post by nathanmarz » Thu, May 26, 2011, 6:59 PM UTC

jbellis wrote: post #24

How is this different from a "traditional" CEP system like Esper? (I mean on the actual processing front, rather than architecturally -- sounds like Storm is a bunch of building blocks instead of a unified system.)