Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Pinterest Architecture Update - 18M Visitors, 10x Growth, 12 Employees, 410 TB

highscalability.com

Post reply on HN
Print view View original

81–88 of 88 posts

Re: Pinterest Architecture Update - 18M Visitors, 10x Growth, 12 Employees, 410 TB

#81

Post by irahul »

edwinnathaniel wrote: post #76
irahul wrote: post #75

Earlier quoted context omitted.

> And the last time you use Java was....5 years ago? > I know Rails and Django and I used cherrypy, cheetah, and sqlalchemy as well, and Java is on par with both modern frameworks if you know to choose the right frameworks. What are these frameworks in Java you talk about which are as expressive as Django and Rails? Less verbose that it was before isn't the same as as expressive as Rails/Django. > In fact, I do not h…

Expressive lies in the language not in the framework, I think you have a different mindset on that one but I'm not sure what you're trying to achieve. Spring MVC is on par with Rails VC (let's leave ActiveRecord for another time) or Django TV minus Django ORM. Testing? Spring has a really good testing library that plays with both straight up Java EE components (Servlet, Portlet, etc). Functional, Integration, Unit-Te…

> Expressive lies in the language not in the framework,

For me, expressiveness is the sum of language expressiveness, framework, culture and api design.

For example, see this crime against humanity

    
    
    
    
        
            You have ${fn:size(emails.unread)} unread email(s)!
        
        
            You have no unread emails!
Now, the designers could have very well implemented:

    You have ${emails.unread ?: 'no'} ${emails.unread?.pluralize('email')} !
> I think you have a different mindset on that one but I'm not sure what you're trying to achieve.

I am not trying to achieve anything, and it's not just a matter of mindset. See my example above. Expressiveness isn't just something which is confined to the language. Horrendous apis and frameworks are a bane for any language.

> Migration? Flyway works nicely and I don't have to spend days to set it up, just less than an hour for multi-year project.

I don't follow your remark that you don't have to spend days to set it up. Which migration lib takes days to set up? I am not contesting there isn't something bad out there(see my JSP example above), but since we are talking Rails/Django, I don't see how it is relevant here.

> JAX-RS can spits both XML and JSON easily with no code changes. I don't have to use Sinatra (or node.js) for web-api and deploy it separately from Rails:

Umm what? Rails needs Sinatra to produce JSON?

    def foo
      respond_to do |format|
        format.html {  }
        format.json {  }
        format.xml { }
      end
    end
> Wait, what am I doing trolling like redditors or slashdotters over mindless debate. > sigh technology has never been the problem, the people are sigh always holds true.

I won't bother responding to this.

Re: Pinterest Architecture Update - 18M Visitors, 10x Growth, 12 Employees, 410 TB

#82

Post by edwinnathaniel »

Fledeskum wrote: post #77
edwinnathaniel wrote: post #73

Earlier quoted context omitted.

When was the last time you use Spring? How about Spring MVC for front-end/controller and or JAX-RS for web API. I find Java to be much more modular and adaptable to different web paradigm: old school MVC, one page JS style (GWT or pure JS), asynchronous (Netty or Servlet 3.0), scalable web api that can spit both xml and json without extra coding (JAX-RS), component oriented (JSF). The integration between business log…

The company I work for (which would qualify as an enterprise by any standards) decided to switch over to Django almost 3 years ago, rewriting our existing Java infrastructure on a project-to-project basis. The reasoning was primarily two-fold, we'd begun to have trouble recruiting good new talent interested in working with Spring/Java and we felt that for what we were doing, Java was not the strongest technology choi…

I find your story to be very interesting. It used to be the case that it is easier to find talents in Java/.NET but that doesn't seem the case these days.

Why Django not Rails since Rails seem to be more popular?

I found it hard to believe that you can switch 55k LoC of Spring MVC + JAX-RS to Django with resulted of 8k. Because from what I've learned over the year, the amount of code requires to write Spring MVC + JAX-RS is very minimum and very close to Rails (my experience is more with Rails) at the very least from the Controller point of view. My personal like of the paradigm is to re-use the code between Spring MVC and JAX-RS. Love it so much.

That is of course if you discount everything else: imports, comments, parentheses, configurations. What about the actual logic?

We typically use JS heavy on the front end and less on JSF/JSP.

But you're right, YMMV. Projects have different requirements and skillset/experience. We typically don't have technical bugs but more of requirement bugs: someone forgot to handle the case of X,Y,Z and as such, it requires about a day or less to plug the issues.

Re: Pinterest Architecture Update - 18M Visitors, 10x Growth, 12 Employees, 410 TB

#83

Post by edwinnathaniel »

irahul wrote: post #81
edwinnathaniel wrote: post #76

Earlier quoted context omitted.

Expressive lies in the language not in the framework, I think you have a different mindset on that one but I'm not sure what you're trying to achieve. Spring MVC is on par with Rails VC (let's leave ActiveRecord for another time) or Django TV minus Django ORM. Testing? Spring has a really good testing library that plays with both straight up Java EE components (Servlet, Portlet, etc). Functional, Integration, Unit-Te…

> Expressive lies in the language not in the framework, For me, expressiveness is the sum of language expressiveness, framework, culture and api design. For example, see this crime against humanity You have ${fn:size(emails.unread)} unread email(s)! You have no unread emails! Now, the designers could have very well implemented: You have ${emails.unread ?: 'no'} ${emails.unread?.pluralize('email')} ! > I think you hav…

http://freemarker.sourceforge.net/ and ditch JSP or use pure JS+HTML.

A typical setup is to use Rails for front-facing website and Sinatra for web-services (RESTful).

What I'm saying is that that can be done with Spring MVC + JAX-RS:

@Produces("application/xml")

@Produces("application/json")

// Want more? add your own? very flexible

// @Produces("application/atom")

// @Produces("text/plain")

// @Produces("application/something+xml") public Employee get(@Parameter("id") final int id){

  // code no need to pepper with respond_to

}

Re: Pinterest Architecture Update - 18M Visitors, 10x Growth, 12 Employees, 410 TB

#84

Post by irahul »

edwinnathaniel wrote: post #83
irahul wrote: post #81

Earlier quoted context omitted.

> Expressive lies in the language not in the framework, For me, expressiveness is the sum of language expressiveness, framework, culture and api design. For example, see this crime against humanity You have ${fn:size(emails.unread)} unread email(s)! You have no unread emails! Now, the designers could have very well implemented: You have ${emails.unread ?: 'no'} ${emails.unread?.pluralize('email')} ! > I think you hav…

http://freemarker.sourceforge.net/ and ditch JSP or use pure JS+HTML. A typical setup is to use Rails for front-facing website and Sinatra for web-services (RESTful). What I'm saying is that that can be done with Spring MVC + JAX-RS: @Produces("application/xml") @Produces("application/json") // Want more? add your own? very flexible // @Produces("application/atom") // @Produces("text/plain") // @Produces("application…

> http://freemarker.sourceforge.net/  and ditch JSP or use pure JS+HTML.

You said frameworks don't decide expressiveness. I was quoting an example that they do. Whether you use jsp or not is tangential. Java isn't anywhere near python or ruby in terms of expressiveness, neither are the frameworks.

EDIT: Like another commenter pointed out, if you can get a 55k LSoC Java project down to 8k Python project, that says something about expressiveness.

Re: Pinterest Architecture Update - 18M Visitors, 10x Growth, 12 Employees, 410 TB

#85

Post by irahul »

Fledeskum wrote: post #77
edwinnathaniel wrote: post #73

Earlier quoted context omitted.

When was the last time you use Spring? How about Spring MVC for front-end/controller and or JAX-RS for web API. I find Java to be much more modular and adaptable to different web paradigm: old school MVC, one page JS style (GWT or pure JS), asynchronous (Netty or Servlet 3.0), scalable web api that can spit both xml and json without extra coding (JAX-RS), component oriented (JSF). The integration between business log…

The company I work for (which would qualify as an enterprise by any standards) decided to switch over to Django almost 3 years ago, rewriting our existing Java infrastructure on a project-to-project basis. The reasoning was primarily two-fold, we'd begun to have trouble recruiting good new talent interested in working with Spring/Java and we felt that for what we were doing, Java was not the strongest technology choi…

> Last year, we switched a 55k loc Spring MVC (with JAX-RS for the json api) over to Django (using tastypie for the api side and celery for offline tasks). The result ended up being little over 8k loc of Python.

While comparing loc, take into account the fact that there were no unknowns - you are simply porting functionality. You generally don't have to solve a problem when you are re-writing a Java codebase in Python, just pythonizing the existing java solution. I would wager if the same system was re-written in Java, it would shave some bloat(provided you are re-implementing and not adding features).

With that said, 55k to 8k is still very impressive.

Re: Pinterest Architecture Update - 18M Visitors, 10x Growth, 12 Employees, 410 TB

#86

Post by Fledeskum »

edwinnathaniel wrote: post #82
Fledeskum wrote: post #77

Earlier quoted context omitted.

The company I work for (which would qualify as an enterprise by any standards) decided to switch over to Django almost 3 years ago, rewriting our existing Java infrastructure on a project-to-project basis. The reasoning was primarily two-fold, we'd begun to have trouble recruiting good new talent interested in working with Spring/Java and we felt that for what we were doing, Java was not the strongest technology choi…

I find your story to be very interesting. It used to be the case that it is easier to find talents in Java/.NET but that doesn't seem the case these days. Why Django not Rails since Rails seem to be more popular? I found it hard to believe that you can switch 55k LoC of Spring MVC + JAX-RS to Django with resulted of 8k. Because from what I've learned over the year, the amount of code requires to write Spring MVC + JA…

Most of the gain was from actual logic, if there had been only models & controllers we wouldn't have seen any major benefits. But there was a lot of IoC with the customary mountains of rather pointless interfaces, an "admin" of sorts & a "messaging" queue running tasks. Most of the POJO data classes ended up being replaced by Python's built in dictionary. Basically the mixture of Python's stdlib, the language ecosystem & Django's got rid of most of our code. In the cases where we couldn't reuse anything from anywhere, something like a business logic class would normally shrink by 2/3. And the frontend was not JS heavy so that factors in.

I hadn't started when the decision to go Django instead of Rails was made (Rails & .NET MVC were also considered) but I've been told it came down to a number of things:

* This is in Europe, generally Ruby is not used very much over here, Python has been used in some shape or form in every company I've have experience of but I've only seen Ruby recently and then mostly for Chef or Puppet.

* Lack of explicit imports did not work in Ruby's favour.

* If needed numpy & scipy don't have a analogue in the Ruby ecosystem.

* General negativity towards the religious zeal permeating through the Rails community.

Re: Pinterest Architecture Update - 18M Visitors, 10x Growth, 12 Employees, 410 TB

#87

Post by jshen »

edwinnathaniel wrote: post #74
irahul wrote: post #66

Earlier quoted context omitted.

> ... and when we look at the um... "old" fart Java... their roster is even more impressive: LinkedIN, Google, NetFlix, Amazon. Sure it is. But I won't use Java for web stuff. I might use it for some background services, but for regular CRUD functionality, Java buys me nothing. I generally use Python(Flask)/Jinja2/Flask-SqlAlchemy. Flask is intuitive; Jinja2 is pleasant and fast; Flask-SQLAlchemy is concise with the…

And the last time you use Java was....5 years ago? Better keep up with the latest frameworks and libraries. I know Rails and Django and I used cherrypy, cheetah, and sqlalchemy as well, and Java is on par with both modern frameworks if you know to choose the right frameworks. In fact, I do not have to deploy multiple processes one for Rails and one for sinatra or node.js. Just some app server or web container and I a…

"I do not have to deploy multiple processes one for Rails and one for sinatra"

FYI, you can mount your sinatra app in your rails app and run them in the same process.

Re: Pinterest Architecture Update - 18M Visitors, 10x Growth, 12 Employees, 410 TB

#88

Post by edwinnathaniel »

Fledeskum wrote: post #86
edwinnathaniel wrote: post #82

Earlier quoted context omitted.

I find your story to be very interesting. It used to be the case that it is easier to find talents in Java/.NET but that doesn't seem the case these days. Why Django not Rails since Rails seem to be more popular? I found it hard to believe that you can switch 55k LoC of Spring MVC + JAX-RS to Django with resulted of 8k. Because from what I've learned over the year, the amount of code requires to write Spring MVC + JA…

Most of the gain was from actual logic, if there had been only models & controllers we wouldn't have seen any major benefits. But there was a lot of IoC with the customary mountains of rather pointless interfaces, an "admin" of sorts & a "messaging" queue running tasks. Most of the POJO data classes ended up being replaced by Python's built in dictionary. Basically the mixture of Python's stdlib, the language ecosyst…

From what I know, JavaEE 6 IoC doesn't require interface so that may cut the code a little bit further. The thing about Java ecosystem is that they rely heavily on IoC/Interface for testing and I can't really blame them because that set-up guarantee super fast unit-tests unlike Rails ActiveRecord (in fact there's a little bit of movement recently in the Rails community that picked up from Java).

ASP.NET MVC won't have much gain over Spring MVC/JAX-RS. In fact, there's nothing similar to JAX-RS in ASP.NET MVC (you'd have to write your own stuff here and there and wire them together, not as straightforward as JAX-RS).

This is an excellent story to share both from technical perspective (Java+Frameworks => Django+Python+Pythonic mindset) and from recruiting perspective. I hope one day you could share (perhaps in slides/presentation style) the actual code and technique in details.

Mind if I ask you where in Europe? (Country/City?) For a while I thought Europe is Rails heavy.

Post reply on HN