Storm - the Hadoop of realtime processing • Hacker News Board

Re: Storm - the Hadoop of realtime processing

#11

Post by nathanmarz » Thu, May 26, 2011, 5:08 PM UTC

maxdemarzi wrote: post #8

" To compute reach, you need to get all the people who tweeted the URL, get all the followers of all those people, unique that set of followers, and then count the number of uniques. It's an intense computation that potentially involves thousands of database calls and tens of millions of follower records." Or you could use a Graph DB to solve a Graph problem. URL -> tweeted_by -> users -> followed_by -> users Try tha…

To do that query on Neo4j, you would need to store in memory on one machine the entire Twitter social graph, all the people who tweeted every URL ever tweeted on Twitter, and then do the computation on a single thread. Neo4j can't handle that scale.

The reach computation on Storm does everything in parallel (across however many machines you need to scale the computation) and gets data using distributed key/value databases (Riak, in our case).

Re: Storm - the Hadoop of realtime processing

#12

Post by justincormack » Thu, May 26, 2011, 5:11 PM UTC

konsl wrote: post #6

ora600 wrote: post #4

Earlier quoted context omitted.

Also, the lack of any scalability charts or diagrams of architecture is suspicious. If you can't make it opensource, at least write a serious paper to support the claims. Like Google did for Big-Table. A lot of people think their systems are scalable and fault-tolerant. Most are not. And from the information provided, we can't tell.

We've released open source projects (most notably ElephantDB and Cascalog) in the past that are successfully used in production by us as well as other companies. You should check them out if you're interested in a measure of quality, though I understand your concern. We're a startup — we're not going to write an academic paper supporting the claims in the post. Nevertheless, Storm's an exciting project many people ar…

Dont call it "the Hadoop of" if it is not open source. Hadoop is notable as an open source project not actually a new idea...

Re: Storm - the Hadoop of realtime processing

#13

Post by nathanmarz » Thu, May 26, 2011, 5:13 PM UTC

vannevar wrote: post #2

Storm sounds great, but this post probably should have waited until it was actually open-sourced. As it is, it just comes across as naked self-promotion based on a technology that could for all we know be vaporware.

(I'm the author of Storm)

Your criticism is totally fair. People have been curious about Storm so we wanted to provide a little bit of information about it. We'll have demos soon, and of course it will be open sourced within a few months.

If you're curious about our credibility, I think our other open source projects speak to the quality of software we produce:

https://github.com/nathanmarz/cascalog https://github.com/nathanmarz/elephantdb

Re: Storm - the Hadoop of realtime processing

#14

Post by Maro » Thu, May 26, 2011, 5:13 PM UTC

I'm not sure if this is the same thing, but there's also a new company called Hadapt (the commercialization of HadoopDB). It's about adapting Hadoop for real-time analytic SQL queries by putting local SQL dbs on the Hadoop nodes and then using the Hadoop plumbing. It's based on Daniel Abadi's research, he's a really smart guy.

Re: Storm - the Hadoop of realtime processing

#15

Post by SeoxyS » Thu, May 26, 2011, 5:24 PM UTC

nathanmarz wrote: post #13

vannevar wrote: post #2

Storm sounds great, but this post probably should have waited until it was actually open-sourced. As it is, it just comes across as naked self-promotion based on a technology that could for all we know be vaporware.

(I'm the author of Storm) Your criticism is totally fair. People have been curious about Storm so we wanted to provide a little bit of information about it. We'll have demos soon, and of course it will be open sourced within a few months. If you're curious about our credibility, I think our other open source projects speak to the quality of software we produce: https://github.com/nathanmarz/cascalog https://github.co…

I think people here are a little too harsh. Storm sounds like an amazing product and I can't wait to play with something like that. Right now, we run a bunch of cron jobs every minute with intense MapReduce queries on mongodb to generate relatively up-to-date analytics. Something like this would be immensely useful. (As well as Mongo's new 2.0 Aggregation pipeline features.)

Now, I agree that it's kind of a bummer we can't play with it right now, but the fact that you guys made this are are going to open source it is already awesome in itself.

Re: Storm - the Hadoop of realtime processing

#16

Post by nphase » Thu, May 26, 2011, 5:31 PM UTC

vannevar wrote: post #2

Storm sounds great, but this post probably should have waited until it was actually open-sourced. As it is, it just comes across as naked self-promotion based on a technology that could for all we know be vaporware.

I'm still waiting on Twitter's rainbird (http://www.slideshare.net/kevinweil/rainbird-realtime-analyt...) to come out!

Re: Storm - the Hadoop of realtime processing

#17

Post by herdrick » Thu, May 26, 2011, 5:48 PM UTC

This sounds great.

This is the traditional realtime processing use case: process messages and update a variety of databases.

Question: I typically think of real-time as a need for user-facing things, i.e. handling a user's requests before he gets bored and goes away. Is Storm set up for that? Or is it mostly meant to update a database with results rather than return them to a waiting process?

Re: Storm - the Hadoop of realtime processing

#18

Post by bmohlenhoff » Thu, May 26, 2011, 5:52 PM UTC

It sounds like a neat project, but I think describing it as "real time" is misleading if you're not also providing information on latency. The majority of the provided use cases seem to indicate a high level of scalability and durability, as well as a high level of throughput, but these are not necessary characteristics of a true real time system.

It's a common misconception. A real time system doesn't have to be fast, efficient, or fault tolerant. A real time system must guarantee with 100% certainty that in all cases it will respond to input X within a time period of Y.

I would be interested to learn the timing issues driving the development of this system and how you've guaranteed such a response time, especially given that it's running on top of the JVM and must therefore deal with a non-deterministic garbage collection process.

Re: Storm - the Hadoop of realtime processing

#19

Post by nathanmarz » Thu, May 26, 2011, 5:56 PM UTC

herdrick wrote: post #17

This sounds great. This is the traditional realtime processing use case: process messages and update a variety of databases. Question: I typically think of real-time as a need for user-facing things, i.e. handling a user's requests before he gets bored and goes away. Is Storm set up for that? Or is it mostly meant to update a database with results rather than return them to a waiting process?

It handles both cases. Storm can be used to asynchronously update databases in realtime in a scalable way (replacing traditional systems of queues and workers). Using Storm for Distributed RPC lets you do intense computations on Storm and return them to a waiting process.

Re: Storm - the Hadoop of realtime processing

#20

Post by scott_s » Thu, May 26, 2011, 6:17 PM UTC

bmohlenhoff wrote: post #18

It sounds like a neat project, but I think describing it as "real time" is misleading if you're not also providing information on latency. The majority of the provided use cases seem to indicate a high level of scalability and durability, as well as a high level of throughput, but these are not necessary characteristics of a true real time system. It's a common misconception. A real time system doesn't have to be fas…

You described a hard real-time system. That exists for things like the controllers on a jet. What's becoming much more prevalent are soft real-time systems that perform analytics. There won't be any catastrophic failure if deadlines aren't meant - and there may not even be any expressed deadline - it's just understood that the data must be processed and analyzed as fast as possible to be useful.