Live data from Hacker News

Python at Netflix

medium.com

201–210 of 250 posts

Re: Python at Netflix

#201
post #184

Earlier quoted context omitted.

As someone who has worked on fully remote game dev volunteer projects for nearly 10 years, it's unbelievable how much the cultural inertia of physical presence has prevented companies from making the rational move of eliminating all possible office space in favor of remote work and skype meetings. So much money wasted on rent, security, and upkeep and so much time (and even lives) lost to commuting when there isn't a…

Even though I would absolutely prefer to work remotely, I have to disagree with you. I find that there are significant benefits to working in the same physical location. It's just way easier to go to someone's desk and ask them something than to have to set up a video chat. Human interaction is just naturally an in person thing. So much is lost if you can only video chat. Sure it may not affect the work directly but…

> It's just way easier to go to someone's desk and ask them something than to have to set up a video chat

That's not a benefit, that's a downside. Interrupting someone while they're working kills productivity. You're far better asking in chat and having someone you're not interrupting help.

Re: Python at Netflix

#202

Earlier quoted context omitted.

Why would you resist it?

It would be nice if the language was typed. Easier for overall maintenance, and refactoring for big projects.

It has a very nice strong typing add on in mypy. Highly recommended for a couple of years now. It’s technically optional but your CI process should take care of that.

Re: Python at Netflix

#203

Id like to contribute to the netflix code base by offering what is obviously a missing but much needed piece of code: def enableAutoPlay( flag ): annoyingAutoPlay = flag return

Afaik you can disable autoplay in the options?

You can disable automatic playback of the next video but can't disable autoplay previews.

Re: Python at Netflix

#204

Earlier quoted context omitted.

> I fail to understand the use of python in a distributed environment while the language has such poor concurrency support Because it's a distributed environment probably is exactly why. Python has (arguably) great concurrency support apart from Multi-threading. https://www.youtube.com/watch?v=MCs5OvhV9S4 So if you need concurrency in the context of a single thread, then Python's GIL is a non-starter. But a distribut…

Even if your app is IO bound, Python's concurrency is painful. Because it's not statically typed, it's too easy to forget an `await` (causing your program to get a Promise[Foo] when you meant to get a Foo) or to overburden your event loop and such things are difficult to debug (we've had several production outages because of these class of bugs). Never mind the papercuts that come about from dealing with the sync/asy…

And that even with async you're still bound by the GIL

Re: Python at Netflix

#205
post #111

I fail to understand the use of python in a distributed environment while the language has such poor concurrency support (on top of the lack of a type system). You can make your application HA, but they are obviously not trying to squeeze out every CPU cycle.

> I fail to understand the use of python in a distributed environment while the language has such poor concurrency support (on top of the lack of a type system). You can make your application HA, but they are obviously not trying to squeeze out every CPU cycle. You're conflating several things that are orthogonal imo. A system can be distributed without concurrency. A concurrent system need not be distributed. Either…

> A system can be distributed without concurrency. A concurrent system need not be distributed.

The only justification for such design I can think of is HA. I.e. you have very light workload that does not saturate a single machine, but nonetheless you create a distributed system with two machines for HA. Such light workload is probably not often the case at Netflix.

Re: Python at Netflix

#206
post #146

Does anyone have more info about this or know of similar projects? We lean on the many of the statistical and mathematical libraries (numpy, scipy, ruptures, pandas) to help automate the analysis of 1000s of related signals when our alerting systems indicate problems. We’ve developed a time series correlation system...

What would you like to know?

push or pull ?

Re: Python at Netflix

#207
post #204

Earlier quoted context omitted.

Even if your app is IO bound, Python's concurrency is painful. Because it's not statically typed, it's too easy to forget an `await` (causing your program to get a Promise[Foo] when you meant to get a Foo) or to overburden your event loop and such things are difficult to debug (we've had several production outages because of these class of bugs). Never mind the papercuts that come about from dealing with the sync/asy…

And that even with async you're still bound by the GIL

Huh? With async, there's typically only ever 1 thread running, so there's no contention for the GIL.

Re: Python at Netflix

#208

Earlier quoted context omitted.

If you’re doing (data analysis|simulations|Image processing) you can offload computation to numpy, which releases the GIL. This allows nice multicore speedups with python and threading. The same holds for various CPU intensive standard library functions implemented in C. The GIL issue is real, but posts like this one confused me for years. Please, don’t exaggerate GIL issues.

Not everything is amenable to numpy and it's pretty easy to make performance worse by throwing numpy at every problem. For example, if your array contains Python objects that are part of an operation, you've likely just introduced a significant performance regression. Worse, there's no way to detect these regressions except to have performance tests. Please, don't understate GIL issues.

Why would you make a NumPy array of objects? Use a list until it makes sense to create an array.

Re: Python at Netflix

#209

Earlier quoted context omitted.

> I fail to understand the use of python in a distributed environment while the language has such poor concurrency support Because it's a distributed environment probably is exactly why. Python has (arguably) great concurrency support apart from Multi-threading. https://www.youtube.com/watch?v=MCs5OvhV9S4 So if you need concurrency in the context of a single thread, then Python's GIL is a non-starter. But a distribut…

Even if your app is IO bound, Python's concurrency is painful. Because it's not statically typed, it's too easy to forget an `await` (causing your program to get a Promise[Foo] when you meant to get a Foo) or to overburden your event loop and such things are difficult to debug (we've had several production outages because of these class of bugs). Never mind the papercuts that come about from dealing with the sync/asy…

Both problems have built-in debug solutions in recent versions of python. The event loop will literally print out all the un-awaited coroutines when it exits, and you can enable debug on the event loop and have it print out every time a coroutine takes longer than a configurable amount of time.

Re: Python at Netflix

#210
post #208

Earlier quoted context omitted.

Not everything is amenable to numpy and it's pretty easy to make performance worse by throwing numpy at every problem. For example, if your array contains Python objects that are part of an operation, you've likely just introduced a significant performance regression. Worse, there's no way to detect these regressions except to have performance tests. Please, don't understate GIL issues.

Why would you make a NumPy array of objects? Use a list until it makes sense to create an array.

Because you’re using Numpy via an intermediate library like pandas and you have composite data in one column?
Post reply on HN