Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Pinterest Architecture Update - 18M Visitors, 10x Growth, 12 Employees, 410 TB

highscalability.com

Post reply on HN
Print view View original

61–70 of 88 posts

Re: Pinterest Architecture Update - 18M Visitors, 10x Growth, 12 Employees, 410 TB

#61

Post by dirtyaura »

pajju wrote: post #57
lionheart wrote: post #16

Earlier quoted context omitted.

Yeah, I'm actually in the process of switching to Python/Django from Rails, so its great to hear how well it scales.

Check this list out, Running Python - Django [1][3][4]: -Instagram[0] -Pinterest -Disqus Disqus.com - Disqus serves over 3 billion page views, and more than 500 million unique visitors a month on it's Django stack. As far as we know we are the largest installation out there. -Mozilla[2] addons.mozilla.com and support.mozilla.com 250k+ add-ons, 150 million views per month, 500+ million api hits per day (firefox checki…

You seem to exaggerate Python's use: gmail is mostly Java, like most of the otger Google apps, LinkedIn ain't using Python to my knowledge.

Re: Pinterest Architecture Update - 18M Visitors, 10x Growth, 12 Employees, 410 TB

#62

Post by skeletonjelly »

moe wrote: post #37
al_james wrote: post #27

Does anyone else feel that 410TB of user data seems quite a lot? If I have my maths right, even if all the 80 million objects are user data (as opposed to, say, logs) thats 5.3mb per object. Considering that most pinterest photos are from the web, that seems quite big.

Yes, that number is ridiculous. For reference, that's about four times bigger than the iTunes music catalog (20mio MP3 files * 5MB average filesize = 100TB).

5MB seems a bit small, they'd also have lossless copies right?

Re: Pinterest Architecture Update - 18M Visitors, 10x Growth, 12 Employees, 410 TB

#63

Post by skeletonjelly »

neovive wrote: post #50

Interesting. I'm migrating some old PHP apps over to Python and have been learning Flask + SQL Alchemy. Why would a site with so much traffic choose a full stack framework like Django that required so much modification?

Because when they started they didn't have the traffic, quicker to get to a running product. Probably easier now to cut back/change Django functionality than rewrite.

Re: Pinterest Architecture Update - 18M Visitors, 10x Growth, 12 Employees, 410 TB

#64

Post by Styck »

edwinnathaniel wrote: post #36
whalesalad wrote: post #31

Earlier quoted context omitted.

There's a kabillion factors to consider though. With that many users, yes the exact implementation matters less and the ancillary systems and how they interact are more critical. But, building the platform, being able to iterate quickly and deliver robust functionality along the way is important as well. Not that Ruby hasn't, but Django and Python have really proved their worth with some of the heavy hitters recently…

... and when we look at the um... "old" fart Java... their roster is even more impressive: LinkedIN, Google, NetFlix, Amazon. Just commenting ... :D

But you're not going to be able to build a Pintrest or a Instagram with 5 people in 2 years with Spring.

Re: Pinterest Architecture Update - 18M Visitors, 10x Growth, 12 Employees, 410 TB

#65

Post by edwinnathaniel »

Styck wrote: post #64
edwinnathaniel wrote: post #36

Earlier quoted context omitted.

... and when we look at the um... "old" fart Java... their roster is even more impressive: LinkedIN, Google, NetFlix, Amazon. Just commenting ... :D

But you're not going to be able to build a Pintrest or a Instagram with 5 people in 2 years with Spring.

You can. I feel sorry to those who can't :)

Re: Pinterest Architecture Update - 18M Visitors, 10x Growth, 12 Employees, 410 TB

#66

Post by irahul »

edwinnathaniel wrote: post #36
whalesalad wrote: post #31

Earlier quoted context omitted.

There's a kabillion factors to consider though. With that many users, yes the exact implementation matters less and the ancillary systems and how they interact are more critical. But, building the platform, being able to iterate quickly and deliver robust functionality along the way is important as well. Not that Ruby hasn't, but Django and Python have really proved their worth with some of the heavy hitters recently…

... and when we look at the um... "old" fart Java... their roster is even more impressive: LinkedIN, Google, NetFlix, Amazon. Just commenting ... :D

> ... and when we look at the um... "old" fart Java... their roster is even more impressive: LinkedIN, Google, NetFlix, Amazon.

Sure it is. But I won't use Java for web stuff. I might use it for some background services, but for regular CRUD functionality, Java buys me nothing. I generally use Python(Flask)/Jinja2/Flask-SqlAlchemy. Flask is intuitive; Jinja2 is pleasant and fast; Flask-SQLAlchemy is concise with the option of exploring the raw power of SQLAlchemy. For regular use cases, nothing in Java beats this combination. I have looked at Play; it comes closer but it's still not there.

Checkout this toy benchmark that was doing the rounds 2 months ago https://github.com/grahamking/Key-Value-Polyglot The naive python solution https://github.com/grahamking/Key-Value-Polyglot/blob/master... performs dog slow. Java has real threads, JIT; but guess what, the naive Java solution still is dog slow https://github.com/grahamking/Key-Value-Polyglot/blob/master...

My specialized python solutions are magnitudes of times faster than both naive Python and Java solutions:

https://github.com/rahulkmr/Key-Value-Polyglot/blob/master/m... https://github.com/rahulkmr/Key-Value-Polyglot/blob/master/m...

The naive solutions were taking around 20 seconds to do 500 writes followed by 500 reads. My changed solution does 5000 writes followed by 5000 reads in under 2 seconds. The 500 and 5000 aren't typos - the naive solutions was taking about 20 seconds for 1000 operations, whereas tailored solution was doing 10000 operations under 2 seconds.

This is the discussion thread http://news.ycombinator.com/item?id=3733090

I get it that this is an IO bound problem, and naturally an epoll based solution will totally smoke a thread-per-request solution, but that's the point I am trying to make. If at the end of the day, intuitive solutions don't work and I have to do custom implementations, I am not getting much out of using Java. I might use it for something which is CPU bound about 70% of the time, but anything other than that, the pains far outweigh benefits.

Re: Pinterest Architecture Update - 18M Visitors, 10x Growth, 12 Employees, 410 TB

#67

Post by irahul »

pajju wrote: post #57
lionheart wrote: post #16

Earlier quoted context omitted.

Yeah, I'm actually in the process of switching to Python/Django from Rails, so its great to hear how well it scales.

Check this list out, Running Python - Django [1][3][4]: -Instagram[0] -Pinterest -Disqus Disqus.com - Disqus serves over 3 billion page views, and more than 500 million unique visitors a month on it's Django stack. As far as we know we are the largest installation out there. -Mozilla[2] addons.mozilla.com and support.mozilla.com 250k+ add-ons, 150 million views per month, 500+ million api hits per day (firefox checki…

EDIT: I upvoted the post, and now it's no longer in gray. So, it was just one downvote.

Why the downvotes? Parent post is just pointing out sites which he/she knows to be using Python and/or Django, some of which are incorrect(Gmail?), or are giving the impression that they run on Python when only small sub-projects are using it(LinkedIn?, Amazon?). There are some mistakes, but on a whole, I don't see anything wrong with someone pointing out something relevant to the discussion, even if it involves bragging about something he is associated with.

Re: Pinterest Architecture Update - 18M Visitors, 10x Growth, 12 Employees, 410 TB

#68

Post by Styck »

edwinnathaniel wrote: post #65
Styck wrote: post #64

Earlier quoted context omitted.

But you're not going to be able to build a Pintrest or a Instagram with 5 people in 2 years with Spring.

You can. I feel sorry to those who can't :)

Of course you can, anything is possible, including convincing yourself Spring is as productive as Rails or Django :)

Nothing against Java, it's the best tool for the job on many occasions but launching quickly and scaling fast are not it's forté.

Re: Pinterest Architecture Update - 18M Visitors, 10x Growth, 12 Employees, 410 TB

#69

Post by raverbashing »

edwinnathaniel wrote: post #56
megaman821 wrote: post #20

Earlier quoted context omitted.

Yes, but the Django ORM helped them launch faster. There is nothing wrong using it to start with and then writing your own SQL queries when you need to scale or even changing your data model to accommodate scaling.

Excuse me but couldn't they just optimize the way they use Django ORM? ... as in: check the output of SQL queries constructed by Django ORM and tweaked it in your Django Model layer such that the resulted SQL query will be tuned for performance? Or at the very least, shouldn't Django ORM at least provide a way to retrieve data with "filtering" as optional parameters? (get only a few columns, but not all..)

Let me tell you something

If you're reading "just tables" without, or with simple relations and conditions, it's ok (because, as you pointed out, that's basically SELECT with)

The problem is when for example you have inheritance, or moderately complex relations.

Then you'll see Django spamming your DB with requests and the DB may be fast but the operation will be slow because of the sheer number of requests.

So yeah, in a sense you can optimize for it, or use SqlAlchemy (which is good, but it has a different philosophy than the Django ORM in the way it's used)

Re: Pinterest Architecture Update - 18M Visitors, 10x Growth, 12 Employees, 410 TB

#70

Post by moe »

skeletonjelly wrote: post #62
moe wrote: post #37

Earlier quoted context omitted.

Yes, that number is ridiculous. For reference, that's about four times bigger than the iTunes music catalog (20mio MP3 files * 5MB average filesize = 100TB).

5MB seems a bit small, they'd also have lossless copies right?

Yes, their total storage consumption is bigger, but the MP3-part of the library is in that ballpark.
Post reply on HN