Twitter: From Ruby on Rails to the JVM [video] • Hacker News Board

Re: Twitter: From Ruby on Rails to the JVM [video]

#51

Post by larrywright » Sat, Jul 30, 2011, 7:02 PM UTC

kemiller wrote: post #24

jefft wrote: post #18

Earlier quoted context omitted.

I recall reading an interview with Evan Weaver that the problem with moving to JRuby was in client libraries they were using, not JRuby itself.

That is indeed the issue if you want to convert an existing rails app to JRuby. But if you're just starting out, it's a different story: you'll end up much more JVM-focused since the best option is generally to use the java equivalent instead of a ruby gem, but there are options. I've been using rails for almost 7 years now (holy crap) and run a medium-sized site on it. It works just fine, but if I were starting from…

Agreed. There are certainly some people who can't use the JVM for some reason (client libraries being a common one), but I think most people would be best served by using JRuby. There's a lot of innovation going on there, and platforms like Torquebox[1] have some compelling benefits.

1. http://torquebox.org/

Re: Twitter: From Ruby on Rails to the JVM [video]

#52

Post by visava » Sat, Jul 30, 2011, 7:03 PM UTC

rrrazdan wrote: post #13

I was recently in a quandary over the choice of technology. I started RoR and I really like it. However I was concerned about long term implications of that choice. The thing that I am taking from this talk is that I shouldn't worry about that, right now. If and when I need to scale, I will have enough resources to make a better choice. Resources that I don't have right now.

try grails http://www.grails.org. Java/groovy version of ROR. It also has a clojure plugin

Re: Twitter: From Ruby on Rails to the JVM [video]

#53

Post by gr3g » Sat, Jul 30, 2011, 7:03 PM UTC

xal wrote: post #16

rrrazdan wrote: post #13

I was recently in a quandary over the choice of technology. I started RoR and I really like it. However I was concerned about long term implications of that choice. The thing that I am taking from this talk is that I shouldn't worry about that, right now. If and when I need to scale, I will have enough resources to make a better choice. Resources that I don't have right now.

Shopify is still 100% ROR and we serve hundreds of millions of requests. You will be fine :-) It's a competitive advantage for us, we move faster then the rest of the market.

Hi Tobi, I was wondering how you guys are doing multi-tenancy.

Re: Twitter: From Ruby on Rails to the JVM [video]

#54

Post by jorgeortiz85 » Sat, Jul 30, 2011, 7:11 PM UTC

troymc wrote: post #34

I noticed that every time Twitter acquired a company, they also accrued a new language: * Summize brought Scala * BackType brought Clojure Has anyone noticed that pattern elsewhere?

Twitter started looking at Scala before the Summize acquisition. Also, to my knowledge, Scala was not being used at Summize.

Re: Twitter: From Ruby on Rails to the JVM [video]

#55

Post by jsavimbi » Sat, Jul 30, 2011, 7:50 PM UTC

lionheart wrote: post #49

jsavimbi wrote: post #48

Earlier quoted context omitted.

> So you can pretty much stick each one on it's own server with it's own database and it'll be fine. That is not a scalable solution. Yes, it'll get you up and running out of the box, but as you keep spinning up servers to host each store and it's database you'll have to keep adding exponential resources (hardware, software, meatware) to the problem and keep you from achieving economies of scale.

Yes, its not an ideal solution for the long term, but its something easy you can do to scale quickly if you have this architecture. It'll make sure your site stays up. Afterwards, you can spend the time doing it properly when things have calmed down a little. That's a lot easier than scaling something like Twitter.

No, sites go down all the time and need to rebooted, reconfigured, redeployed, strangled, etc. by a human who has to watch the servers 24 hours a day. I'm ignorant to the number of customers that Shopify is hosting these stores for, but let's say for example that one human can monitor 100 virtual machines during an eight-hour shift and you have 500 vm's running at the same time over a series of physical servers. That means, in HR terms, you'll need five people per shift or fifteen per day to monitor and act upon the vm's, along with at least two operations people, one of whom will be carrying the pager for the entire 24-hour period. That's seventeen people needed to run your operation not including sick/vacation/leave time that needs to be covered. In salary costs alone, that's over $1million/yr., not including recruiting and management salary, benefits and compensation. And does not take into effect any development, real estate, office essentials, hardware and software costs.

> Afterwards, you can spend the time doing it properly when things have calmed down a little.

That'll cost you an extra million dollars to develop and deploy while simultaneously running your existing operations and migrating your clients over to the new solution.

It doesn't scale.

Re: Twitter: From Ruby on Rails to the JVM [video]

#56

Post by moe » Sat, Jul 30, 2011, 7:55 PM UTC

I'm still not getting the futz they're making over their "scale".

So your inbound load is 7000 tweets/sec or roughly 250 MBit/s (assuming 4k per tweet). Then you fan that out to (assuming) 20 append-only mailboxes on average.

Perhaps my assumptions are far off, but I'm only arriving at a couple GBit/s here and a low two digit number of terabytes/storage per year.

This sounds like "a couple racks" to me, not like "a couple datacenters".

Re: Twitter: From Ruby on Rails to the JVM [video]

#57

Post by gnuvince » Sat, Jul 30, 2011, 7:58 PM UTC

moe wrote: post #56

I'm still not getting the futz they're making over their "scale". So your inbound load is 7000 tweets/sec or roughly 250 MBit/s (assuming 4k per tweet). Then you fan that out to (assuming) 20 append-only mailboxes on average. Perhaps my assumptions are far off, but I'm only arriving at a couple GBit/s here and a low two digit number of terabytes/storage per year. This sounds like "a couple racks" to me, not like "a c…

As I understand it -- and I might be wrong here -- the inbound traffic is not the real problem; it's the distribution to all the followers. If you have 7000 tweets coming in and you need to send it to the 120 followers of all these posters in real time, that makes a lot of data shuffling.

Re: Twitter: From Ruby on Rails to the JVM [video]

#58

Post by mark_l_watson » Sat, Jul 30, 2011, 8:12 PM UTC

sahglie wrote: post #27

With the release of Java 7 (invokedynamic) the performance of these dynamic languages (like ruby and python) may become much less of a factor (JRuby and Jython). At least that's what the JRuby folks imply: http://www.engineyard.com/blog/2011/jruby-1-6-released-now-w... Punch Line from Link: "There’s a very real chance that invokedynamic could improve JRuby performance many times, putting us on par with our statically…

Right on! I was going to make the same comment until I saw yours. Charles Nutter has been very enthusiastic about invokedynamic based speed improvements - can't wait.

A little off topic: as a consultant it seems like the demand for Clojure was been tremendous: Clojure is a nice language and very performant. It will be really interesting to see how much large speed improvements in JRuby will cut into Java's, Clojure's and Scala's developer market-share.

Re: Twitter: From Ruby on Rails to the JVM [video]

#59

Post by petercooper » Sat, Jul 30, 2011, 8:16 PM UTC

sscheper wrote: post #35

rhizome wrote: post #21

Earlier quoted context omitted.

OP is an ontwik posting bot. Thanks.

Surprised no other person brought this up.

Ontwik clearly provides some sort of service, if only to dredge up YouTube videos that we've otherwise missed.. but I can't help but feel there's a "better way" for these videos to be discovered than a site that just embeds and adds no editorial context.

Re: Twitter: From Ruby on Rails to the JVM [video]

#60

Post by psykotic » Sat, Jul 30, 2011, 8:16 PM UTC

jsavimbi wrote: post #48

lionheart wrote: post #38

Earlier quoted context omitted.

Well, correct me if I'm wring but Shopfiy is a completely different scaling problem from Twitter. As I understand it Shopify's individual hosted stores are pretty much self-contained. So you can pretty much stick each one on it's own server with it's own database and it'll be fine. Twitter accounts all have to be able to talk to eachother in real time so you can't do that. My current startup has a Shopify-like archit…

> So you can pretty much stick each one on it's own server with it's own database and it'll be fine. That is not a scalable solution. Yes, it'll get you up and running out of the box, but as you keep spinning up servers to host each store and it's database you'll have to keep adding exponential resources (hardware, software, meatware) to the problem and keep you from achieving economies of scale.

>Yes, it'll get you up and running out of the box, but as you keep spinning up servers to host each store and it's database you'll have to keep adding exponential resources (hardware, software, meatware) to the problem and keep you from achieving economies of scale.

No, you'd be adding resources at a _linear_ rate relative to the growth of the customer base. The point about economies of scale is true enough but has nothing to do with a lack of exponential growth in costs.