Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Python at Netflix

medium.com

Post reply on HN
Print view View original

241–250 of 250 posts

Re: Python at Netflix

#241

Post by rifung »

driverdan wrote: post #201
rifung wrote: post #184

Earlier quoted context omitted.

Even though I would absolutely prefer to work remotely, I have to disagree with you. I find that there are significant benefits to working in the same physical location. It's just way easier to go to someone's desk and ask them something than to have to set up a video chat. Human interaction is just naturally an in person thing. So much is lost if you can only video chat. Sure it may not affect the work directly but…

> It's just way easier to go to someone's desk and ask them something than to have to set up a video chat That's not a benefit, that's a downside. Interrupting someone while they're working kills productivity. You're far better asking in chat and having someone you're not interrupting help.

I'm sorry that you feel that way. I consider being able to help another person to be a huge privilege.

What you describe sounds like it's putting one person's progress over another. Sure a mentor may be disrupted but by unblocking someone else they're still increasing the overall productivity of the team and thus is a net gain.

> You're far better asking in chat and having someone you're not interrupting help.

This only works when your question is general enough that many people can help you

Re: Python at Netflix

#242

Post by MaxBarraclough »

gvd wrote: post #111

I fail to understand the use of python in a distributed environment while the language has such poor concurrency support (on top of the lack of a type system). You can make your application HA, but they are obviously not trying to squeeze out every CPU cycle.

> poor concurrency support

Poor intra-process parallelism, yes. Python is pretty capable when it comes to concurrency.

Re: Python at Netflix

#243

Post by mixmastamyk »

weberc2 wrote: post #240
mixmastamyk wrote: post #237

Earlier quoted context omitted.

They is Netflix, and other post-production oriented users. You know, what this article and discussion is about?

It's not at all obvious that "they" refers to "netflix and other post-production oriented users", and your argument is a tautology "Python is good at the things that Python is good at". Obviously. The rest of us are debating what those things are or are not.

The subject is well-trodden, there's not much to debate. Python is not good at threading, but works well in multiprocessing situations. Netflix is using it in the later situation, and not the former. Async is unlikely to be a use case either.

Re: Python at Netflix

#244

Post by relix42 »

Tycho wrote: post #195
aaronblohowiak wrote: post #183

Earlier quoted context omitted.

What would you like to know?

The correlations part specifically, what does it do, what’s the modelling approach, and is there more info about it anywhere?

Initially we wrote the library to help us answer the question about what change may be causing impact to our customers' experiences. Out of the billions of time series metrics we have we knew there were about 10,000 or so likely (for this initial use case), possible candidates and we wanted to reduce that set as far as possible and either (1) find /the/ candidate for problem or (2) produce a short list of things for the humans to look into. In order to get to the point where we could ensure high likelihood of apropos correlations, we needed to do some work on the signals first.

First we detect if any of the possible candidate signals were born or died in the interesting time period. We use these time points to reduce the window we'll use to pass to the correlation functions. We can also detect any changepoints in the time series and apply similar logic. Once we've determined the best window bounds for each candidate signal, we use pearson and spearman correlation functions to get a score for the pair of signals -- the initial signal that started the inquiry and the candidate signal using the determined time window.

The code is about 98% data preparation, signal analysis, and window determination and about 2% correlation work.

(I've tried to summarize quite a bit, let me know if you'd like clarifications or have other questions.)

Re: Python at Netflix

#245

Post by wutbrodo »

gonehome wrote: post #133
wutbrodo wrote: post #110

Earlier quoted context omitted.

> The other way is already working at a famous company and having another famous company send you a recruitment email via linked in, but that typically requires you to have done the referral way first. Just as a single data point, the last clause here isn't true in my experience. I worked at Google for a few years some years ago, and I get a regular flow of recruiter spam from all the big companies for Bay Area roles…

Yeah, but you had to work at google first. I wasn’t clear, but I meant in order to already be working at a famous company you probably already had to figure out the referral piece at least once.

Ah I see, I misunderstood the final clause as stating that you need a referral _as_ an ex-Googler etc. Thanks for clarifying!

Re: Python at Netflix

#246

Post by xapata »

weberc2 wrote: post #215
xapata wrote: post #211

Earlier quoted context omitted.

Ah. Pandas is the problem. Unfortunately, you need to understand how its features are implemented to use it well. Still, my main trouble with Pandas is unnecessary memory bloat, not compute inefficiency. That caveat is somewhat true for all programming abstractions, but well-designed interfaces make the more efficient techniques more obvious and beautiful, while the inefficient or risky techniques are made esoteric a…

Pandas isn't the problem, the problem is assuming that "$LIBRARY releases the GIL so things will be fast!". It's a pennywise, pound-foolish approach to performance. Someone will write a function assuming the user is only going to pass in a list of ints and someone else will extend that function to take a list of Tuple[str, int] or something and all of a sudden your program has a difficult to debug performance regress…

> Someone will write a function assuming the user is only going to pass in a list of ints and someone else will extend that function to take a list of Tuple[str, int]

There are plenty of pitfalls in leaky abstractions. Establishing that fast numeric calculations only work with specific numeric types seems to help.

One thing you seem to be encountering, that I've seen a few times, is that people don't realize NumPy and core Python are almost orthoganal. The best practices for each are nearly opposite. I try to make it clear when I'm switching from one to the other by explaining the performance optimization (broadly) in comments.

Regardless, any function that receives a ``list`` of ints will need to convert to an ndarray if it wants NumPy speed. If the function interface is modified later, I think it's fair to expect the editor to understand why.

Re: Python at Netflix

#247

Post by weberc2 »

xapata wrote: post #246
weberc2 wrote: post #215

Earlier quoted context omitted.

Pandas isn't the problem, the problem is assuming that "$LIBRARY releases the GIL so things will be fast!". It's a pennywise, pound-foolish approach to performance. Someone will write a function assuming the user is only going to pass in a list of ints and someone else will extend that function to take a list of Tuple[str, int] or something and all of a sudden your program has a difficult to debug performance regress…

> Someone will write a function assuming the user is only going to pass in a list of ints and someone else will extend that function to take a list of Tuple[str, int] There are plenty of pitfalls in leaky abstractions. Establishing that fast numeric calculations only work with specific numeric types seems to help. One thing you seem to be encountering, that I've seen a few times, is that people don't realize NumPy an…

> There are plenty of pitfalls in leaky abstractions

Sure, but this is a _massive_ pitfall. It's an optimization that can trivially make your code slower than the naive Python implementation, all due to a leaky abstraction.

> Regardless, any function that receives a ``list`` of ints will need to convert to an ndarray if it wants NumPy speed. If the function interface is modified later, I think it's fair to expect the editor to understand why.

Yeah, that was a toy example. In practice, the scenario was similar except the function called to a third party library that used Numpy under the hood. We introduced a ton of complexity to use this third party library on the grounds that "it will make things fast" instead of the naive list implementation, and the very next sprint we needed to update it such that it became 10X slower than the naive Python implementation.

That's the starkest example, but there have been others and there would have been many more if we didn't have the stark example to point to.

The current slogan is "just use Python; you can always make things fast with Numpy/native code!", but it should be "use Python if you have a deep understanding of how Numpy (or whatever native library you're using) makes things fast such that you can count on that invariant to hold even as your requirements change" or some such.

Re: Python at Netflix

#248

Post by weberc2 »

mixmastamyk wrote: post #243
weberc2 wrote: post #240

Earlier quoted context omitted.

It's not at all obvious that "they" refers to "netflix and other post-production oriented users", and your argument is a tautology "Python is good at the things that Python is good at". Obviously. The rest of us are debating what those things are or are not.

The subject is well-trodden, there's not much to debate. Python is not good at threading, but works well in multiprocessing situations. Netflix is using it in the later situation, and not the former. Async is unlikely to be a use case either.

> The subject is well-trodden, there's not much to debate

And yet we see the same incorrect information trotted out over and over again.

Re: Python at Netflix

#249

Post by xapata »

weberc2 wrote: post #247
xapata wrote: post #246

Earlier quoted context omitted.

> Someone will write a function assuming the user is only going to pass in a list of ints and someone else will extend that function to take a list of Tuple[str, int] There are plenty of pitfalls in leaky abstractions. Establishing that fast numeric calculations only work with specific numeric types seems to help. One thing you seem to be encountering, that I've seen a few times, is that people don't realize NumPy an…

> There are plenty of pitfalls in leaky abstractions Sure, but this is a _massive_ pitfall. It's an optimization that can trivially make your code slower than the naive Python implementation, all due to a leaky abstraction. > Regardless, any function that receives a ``list`` of ints will need to convert to an ndarray if it wants NumPy speed. If the function interface is modified later, I think it's fair to expect the…

I have mixed feelings about your conclusion. On one hand I don't want to discourage newbies from using Python. On the other, I enjoy that my expertise is valuable.

It seems reasonable that different parts of the code are appropriate for modification by engineers of differing skills.

Re: Python at Netflix

#250

Post by IanCal »

jimmaswell wrote: post #180
IanCal wrote: post #102

Earlier quoted context omitted.

> So much money wasted on rent, Although this now translates to a cost on me needing an extra room in the house for an office.

I do fine using my normal desk. Open AWS and Skype at the start of the day, do work, close them at 5, that's it. Bring a laptop to a different corner of the house or a library/coffee shop if you can't adapt to using your personal space for work. Working outside is nice when the weather permits.

If I didn't work from home, I'd not have a room for an office. Working from home means I need an extra room realistically.
Post reply on HN