Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Python at Netflix

techblog.netflix.com

Post reply on HN
Print view View original

61–70 of 78 posts

Re: Python at Netflix

#61

Post by pwang »

arpineh wrote: post #3
jedberg wrote: post #2

As one of the main drivers of Python at Netflix, this one is near and dear to my heart. Let me know if you have any questions, or come visit our booth!

Hi. There was many interesting things listed at the blog post. I have been planning a system similar in idea to Sting, your OLAP server (?). I'm planning a light RPC style system on top of Pandas dataframe objects, transferring HTTP requests to Pandas method calls. Memory usage of such a system with our data is too expensive if all data is in use. So, I'm investigating possible caching and database solutions. Could y…

Have you looked at the Cubes project?

http://databrewery.org/cubes.html

Re: Python at Netflix

#62

Post by pwang »

jedberg wrote: post #44
jkldotio wrote: post #39

Python really does hit a sweet-spot for many types of operations. Devops capacity in system scripting, crawling and scraping with excellent web frameworks (Django through to Bottle). On top of all that there are great machine learning libraries as well as good APIs to almost anything not written in Python. My site gets huge amounts of NLP, machine learning and information extraction done in just a few lines because o…

You know, it never occurred to me to use iPython as my shell. I think that would make me more productive. I'll have to try that, thanks!

I think someone asked the question at a Scipy conference a few years back, "Who here uses IPython as their primary shell", and IIRC about 30% of the people raised their hands.

Re: Python at Netflix

#63

Post by pwang »

daeken wrote: post #51
jedberg wrote: post #27

Earlier quoted context omitted.

I like Python for the speed of development. Also, at least for these internal apps, speed is generally of secondary concern, since these aren't typically high load applications. At reddit we solved the problem by writing the most often called parts in C and using c extensions. There is definitely more work to do there, but overall I think Python gets a bad rap as far as speed is concerned. It's certainly not the quic…

> At reddit we solved the problem by writing the most often called parts in C and using c extensions. I'm wondering if anyone has solved these sorts of problems with Cython in the real world. It looks like a great solution for a large set of problems in terms of Python performance, since it becomes a CPython extension, but I never hear anyone talk about it. Can anyone chime in?

Cython is used all over the place in some parts of the real world, but you may not hear about those parts because they are not that connected to web app development.

The entire scientific Python ecosystem has basically moved on to Cython. Its use is quite prevalent, and it's basically displacing SWIG as the preferred wrapping tool of choice.

Re: Python at Netflix

#64

Post by jedberg »

bryanwb wrote: post #54
jedberg wrote: post #2

As one of the main drivers of Python at Netflix, this one is near and dear to my heart. Let me know if you have any questions, or come visit our booth!

would you be interested to talk about these tools on the http://foodfightshow.org devops podcast? we would love to discuss them with you!

I would, but I could probably find you people who know more about the tools, like the folks who wrote them. :)

Hit me up on email (it's in my profile).

Re: Python at Netflix

#65

Post by rdtsc »

tootie wrote: post #53
frankwiles wrote: post #37

Earlier quoted context omitted.

Ummm, not sure where you're getting your information but Python has threads ( http://docs.python.org/2/library/threading.html ), first-class functions ( http://en.wikipedia.org/wiki/First-class_function ), and dependency management tools (google for pip, easy_install, etc.)

Sorry, I should have said "lack of true threading in CPython" by which I mean the GIL limitation. And lambda expressions are lame compared to what you can do in JavaScript/Perl/Scala. And the build tools suck compared to Maven which is saying a lot. Maven is cross-platform and install-free. It takes a long time to makes head and tails of pip, easy_install and virtualenv and it still does less than Maven.

What do you use "true" threading for? Are you multiplying matrices? If so use Numpy. Are you downloading data and accessing database? If so, you threading, threading in Python works great for that. If you hit memory size issues, switch to gevent or eventlet for green threads.

I have been building large projects in Python for the last 10 yeas and never really thought "Hmm I wish I had more powerful lambda functions". You can always just have regular function and pass that around. Why do you need lambdas so much?

Re: Python at Netflix

#66

Post by rdtsc »

jedberg wrote: post #28
voidlogic wrote: post #7

As a Java/Python shop I think Netflix should evaluate Go after 1.1 RC comes out in April. Go is very popular with people who have heavily used both Java and Python (as well as C).

I believe we do have one or two people here looking at Go. Given our culture, there really is nothing stopping anyone from using it, other than the fact that they would have to figure out how to interact with the platform.

Anyone used or uses Erlang?

Re: Python at Netflix

#67

Post by jamiltron »

tootie wrote: post #53
frankwiles wrote: post #37

Earlier quoted context omitted.

Ummm, not sure where you're getting your information but Python has threads ( http://docs.python.org/2/library/threading.html ), first-class functions ( http://en.wikipedia.org/wiki/First-class_function ), and dependency management tools (google for pip, easy_install, etc.)

Sorry, I should have said "lack of true threading in CPython" by which I mean the GIL limitation. And lambda expressions are lame compared to what you can do in JavaScript/Perl/Scala. And the build tools suck compared to Maven which is saying a lot. Maven is cross-platform and install-free. It takes a long time to makes head and tails of pip, easy_install and virtualenv and it still does less than Maven.

Lambda expressions are not necessarily first-class functions (I mean, they are in Python, but so is any other old function).

First class functions simply mean that a language treats functions as it would nearly any other data type, namely that they may be passed as parameters, bound to variables, and returned from other functions.

Python's poor handling of lambda expressions do not invalidate that functions are first-class citizens.

Re: Python at Netflix

#68

Post by jedberg »

rdtsc wrote: post #66
jedberg wrote: post #28

Earlier quoted context omitted.

I believe we do have one or two people here looking at Go. Given our culture, there really is nothing stopping anyone from using it, other than the fact that they would have to figure out how to interact with the platform.

Anyone used or uses Erlang?

I don't think so. I suppose it would be possible, but it might be hard to interact with the platform.

Re: Python at Netflix

#69

Post by jasongrout »

takluyver wrote: post #58
daeken wrote: post #51

Earlier quoted context omitted.

> At reddit we solved the problem by writing the most often called parts in C and using c extensions. I'm wondering if anyone has solved these sorts of problems with Cython in the real world. It looks like a great solution for a large set of problems in terms of Python performance, since it becomes a CPython extension, but I never hear anyone talk about it. Can anyone chime in?

I get the impression that Cython is used quite a bit in the scientific computing side of Python, but not so much for web development. pandas & pyzmq are two projects I know that make serious use of Cython.

Sage [1] (which you might say is the birthplace of Cython) makes a very serious use of Cython as well.

[1] http://sagemath.org

Re: Python at Netflix

#70

Post by elq »

voidlogic wrote: post #7

As a Java/Python shop I think Netflix should evaluate Go after 1.1 RC comes out in April. Go is very popular with people who have heavily used both Java and Python (as well as C).

I've used go at netflix before to hack up a quick load test, but recently the vast majority of my non java/c++ code at netflix has been python for writing glue to interact with our ops infrastructure and julia, which recently supplanted python/pandas/scikit, for my my research oriented stuff.

But overall go is a really neat "systems" level language.

Post reply on HN