Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Storm - the Hadoop of realtime processing

tech.backtype.com

Post reply on HN
Print view View original

31–40 of 47 posts

Re: Storm - the Hadoop of realtime processing

#31

Post by nathanmarz »

herdrick wrote: post #27
nathanmarz wrote: post #11

Earlier quoted context omitted.

To do that query on Neo4j, you would need to store in memory on one machine the entire Twitter social graph, all the people who tweeted every URL ever tweeted on Twitter, and then do the computation on a single thread. Neo4j can't handle that scale. The reach computation on Storm does everything in parallel (across however many machines you need to scale the computation) and gets data using distributed key/value data…

Nathan, we'd love to hear your postmortem on BackType's experience with Neo4J, and how Sphinx is turning out.

We used Neo4j over a year ago, and it was pretty unstable when we used it. The database files were getting corrupted pretty frequently (a few times a week), so it just didn't work out for us. Ultimately it was for a small feature, so rather than continue to struggle with Neo4j we just reimplemented the feature using Sphinx. Like I said, that was a long time ago and Neo4j may have gotten a lot better since then.

Re: Storm - the Hadoop of realtime processing

#32

Post by scott_s »

tlrobinson wrote: post #28
scott_s wrote: post #26

I work on a similar system that was previously discussed on HN: http://news.ycombinator.com/item?id=2442977

Watson? Is that the right link?

It's the right link, but the article is poorly written. See my corrections in the comments.

Re: Storm - the Hadoop of realtime processing

#33

Post by nathanmarz »

earl wrote: post #30

How is this different / better than Yahoo S4 [1], which does have code on github? [2]? Why did you choose to build this, or did you start before S4 became public? [1] http://docs.s4.io/ [2] https://github.com/s4/core

Great question. S4 came out right around the time we started designing Storm.

The projects share similarities. The biggest difference with S4 is that Storm guarantees that messages will be processed, whereas S4 will just drop the messages. Getting this level of reliability is more than just using TCP to send messages - you need to track the processing of messages in an efficient way and retry messages if the message doesn't get completed for some reason (like a node goes down). Implementing reliability is non-trivial and affects the design of the whole system.

We also felt that there was a lot of accidental complexity in S4's API, but that's a secondary issue.

Re: Storm - the Hadoop of realtime processing

#34

Post by ScottBurson »

TW;DR!!

For a variety of reasons, I keep my browser windows about 900 pixels wide. Your site requires a honking 1280 to get rid of the horizontal scrollbar -- and can't be read in 900 without scrolling horizontally for every line (i.e. the menu on the left is much too wide).

(OT, I know, but it's a pet peeve of mine. It's been known for years how to use CSS to make pages stretch or squish, within reason, to the user's window width. 900 is not too narrow!)

EDITED to add: yeah, I'm willing to spend some karma points on this, if that's what happens. Wide sites are getting more common, and this is one of the worst I've seen.

Re: Storm - the Hadoop of realtime processing

#35

Post by matclayton »

nphase wrote: post #16
vannevar wrote: post #2

Storm sounds great, but this post probably should have waited until it was actually open-sourced. As it is, it just comes across as naked self-promotion based on a technology that could for all we know be vaporware.

I'm still waiting on Twitter's rainbird ( http://www.slideshare.net/kevinweil/rainbird-realtime-analyt... ) to come out!

Me too, I asked them the other day about it,

Response - http://twitter.com/#!/kevinweil/status/73263430873792512

Re: Storm - the Hadoop of realtime processing

#37

Post by herdrick »

nathanmarz wrote: post #31
herdrick wrote: post #27

Earlier quoted context omitted.

Nathan, we'd love to hear your postmortem on BackType's experience with Neo4J, and how Sphinx is turning out.

We used Neo4j over a year ago, and it was pretty unstable when we used it. The database files were getting corrupted pretty frequently (a few times a week), so it just didn't work out for us. Ultimately it was for a small feature, so rather than continue to struggle with Neo4j we just reimplemented the feature using Sphinx. Like I said, that was a long time ago and Neo4j may have gotten a lot better since then.

OK, thanks! That's valuable info.

Re: Storm - the Hadoop of realtime processing

#38

Post by nathanmarz »

pnathan wrote: post #36

Can you comment on distributing non-JAR software? Also, this sounds faintly like the old SunGridEngine.

If you're using a programming language other than Java with Storm, you simply include those files in the jar that you submit to the cluster. For example, you would put your Python processing code "component.py" in the resources/ subdirectory in your jar (A jar is basically just a zipfile).

Re: Storm - the Hadoop of realtime processing

#39

Post by yummyfajitas »

SeoxyS wrote: post #15
nathanmarz wrote: post #13

Earlier quoted context omitted.

(I'm the author of Storm) Your criticism is totally fair. People have been curious about Storm so we wanted to provide a little bit of information about it. We'll have demos soon, and of course it will be open sourced within a few months. If you're curious about our credibility, I think our other open source projects speak to the quality of software we produce: https://github.com/nathanmarz/cascalog https://github.co…

I think people here are a little too harsh. Storm sounds like an amazing product and I can't wait to play with something like that. Right now, we run a bunch of cron jobs every minute with intense MapReduce queries on mongodb to generate relatively up-to-date analytics. Something like this would be immensely useful. (As well as Mongo's new 2.0 Aggregation pipeline features.) Now, I agree that it's kind of a bummer we…

As a happy user of ElephantDB, I'd say people are definitely too harsh. Elephantdb is awesome - my company has completely replaced HBase with ElephantDB and MaryJane (a lightweight way of putting data into hadoop that we wrote, https://github.com/stucchio/MaryJane- ).

That said, I'd love to see some code released, even if it isn't ready for primetime.

Re: Storm - the Hadoop of realtime processing

#40

Post by iramiller »

ScottBurson wrote: post #34

TW;DR!! For a variety of reasons, I keep my browser windows about 900 pixels wide. Your site requires a honking 1280 to get rid of the horizontal scrollbar -- and can't be read in 900 without scrolling horizontally for every line (i.e. the menu on the left is much too wide). (OT, I know, but it's a pet peeve of mine. It's been known for years how to use CSS to make pages stretch or squish, within reason, to the user'…

I had to head back and check the article out again since I had not noticed the width being an issue. For what it is worth it scaled down and was perfectly usable on my iPad.

I find your width comments especially relevant to me right because we have started a new site design project focusing on offering 5 different width based layouts. Your comment is proof that offering multiple content width options to desktop users and not just mobile is useful to some people besides just me.

Post reply on HN