Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Python at Netflix

medium.com

Post reply on HN
Print view View original

211–220 of 250 posts

Re: Python at Netflix

#211

Post by xapata »

weberc2 wrote: post #210
xapata wrote: post #208

Earlier quoted context omitted.

Why would you make a NumPy array of objects? Use a list until it makes sense to create an array.

Because you’re using Numpy via an intermediate library like pandas and you have composite data in one column?

Ah. Pandas is the problem. Unfortunately, you need to understand how its features are implemented to use it well. Still, my main trouble with Pandas is unnecessary memory bloat, not compute inefficiency.

That caveat is somewhat true for all programming abstractions, but well-designed interfaces make the more efficient techniques more obvious and beautiful, while the inefficient or risky techniques are made esoteric and ugly.

Re: Python at Netflix

#212

Post by weberc2 »

jnwatson wrote: post #209
weberc2 wrote: post #167

Earlier quoted context omitted.

Even if your app is IO bound, Python's concurrency is painful. Because it's not statically typed, it's too easy to forget an `await` (causing your program to get a Promise[Foo] when you meant to get a Foo) or to overburden your event loop and such things are difficult to debug (we've had several production outages because of these class of bugs). Never mind the papercuts that come about from dealing with the sync/asy…

Both problems have built-in debug solutions in recent versions of python. The event loop will literally print out all the un-awaited coroutines when it exits, and you can enable debug on the event loop and have it print out every time a coroutine takes longer than a configurable amount of time.

> The event loop will literally print out all the un-awaited coroutines when it exits

IIRC, I've only ever seen "unawaited coroutine found" (or similar) errors; I've never seen anything that points to a specific unawaited coroutine. In either case, a bug in prod is still many times worse than compile time type error.

> you can enable debug on the event loop and have it print out every time a coroutine takes longer than a configurable amount of time

I don't run my production servers in debug mode, and even when I do manage to find the problem, I have limited options for solving it. Usually it amounts to refactoring out the offending code into a separate process or service.

An extreme counterpoint is a language like Go which

1) Is roughly 100X faster in single-threaded, CPU-bound execution anyway

2) Allows for additional optimizations that simply aren't possible in Python (mostly involving reduced allocations and improved cache coherence)

3) Has a runtime that balances CPU load across all available cores

This isn't a "shit on Python" post; only that concurrency really isn't Python's strong suit (yet).

Re: Python at Netflix

#213

Post by drugme »

jimmaswell wrote: post #180
IanCal wrote: post #102

Earlier quoted context omitted.

> So much money wasted on rent, Although this now translates to a cost on me needing an extra room in the house for an office.

I do fine using my normal desk. Open AWS and Skype at the start of the day, do work, close them at 5, that's it. Bring a laptop to a different corner of the house or a library/coffee shop if you can't adapt to using your personal space for work. Working outside is nice when the weather permits.

I do fine using my normal desk.

Not everyone has space in their apartment for a "normal desk".

Bring a laptop to a different corner of the house or a library/coffee shop if you can't adapt to using your personal space for work.

Not every coffee shop likes having people lounging around for 8 or more hours a day, making minimal purchase. Aside from the fact that most are too noisy to concentrate, have shit wifi, bathrooms that are frequently occupied / barely work, etc.

Re: Python at Netflix

#214

Post by gonehome »

ohithereyou wrote: post #198
gonehome wrote: post #86

Earlier quoted context omitted.

The only way to get hired at competitive companies is via referral - if you live in the Bay Area you probably know someone or know someone who knows someone at Netflix. Trying any other way is a lottery - and if you didn’t go to MIT/Stanford/CalTech/Harvard then your odds are pretty bad. The other way is already working at a famous company and having another famous company send you a recruitment email via linked in,…

The FAANGs support diversity. You can go to any of MIT, Stanford, CalTech, or Harvard.

I know this was just an off the cuff response, but it's a little more nuanced than that.

They don't really care where you went or even if you have a degree, but there are so many people applying through the front door that that ends up being a filter everyone uses. There are obviously excellent people outside of that, but it's harder to find them through the noise.

This is another reason I think Lamda School is great - they're actually attacking this problem too: http://zalberico.com/essay/2019/04/08/lambda-school.html

Re: Python at Netflix

#215

Post by weberc2 »

xapata wrote: post #211
weberc2 wrote: post #210

Earlier quoted context omitted.

Because you’re using Numpy via an intermediate library like pandas and you have composite data in one column?

Ah. Pandas is the problem. Unfortunately, you need to understand how its features are implemented to use it well. Still, my main trouble with Pandas is unnecessary memory bloat, not compute inefficiency. That caveat is somewhat true for all programming abstractions, but well-designed interfaces make the more efficient techniques more obvious and beautiful, while the inefficient or risky techniques are made esoteric a…

Pandas isn't the problem, the problem is assuming that "$LIBRARY releases the GIL so things will be fast!". It's a pennywise, pound-foolish approach to performance. Someone will write a function assuming the user is only going to pass in a list of ints and someone else will extend that function to take a list of Tuple[str, int] or something and all of a sudden your program has a difficult to debug performance regression.

In general, the "just rewrite the slow parts in C!" motto is terrible advice because it's unlikely that it will actually make your code appreciably slower, and if it does, it's very likely to be defeated unexpectedly as soon as requirements change. Using FFI to make things faster can work, but only if you've really considered your problem and you're quite sure you can safely predict relevant changes to requirements.

Re: Python at Netflix

#216

Post by philwelch »

mixmastamyk wrote: post #73
inglor wrote: post #68

Earlier quoted context omitted.

I beg to differ, I've seen it adopted in quite a few places by large companies. I've rarely seen it used in startups though. My evidence is anecdotal (so is yours though). I definitely agree that it's not seeing nearly as much adoption as let's say... TypeScript in the JavaScript ecosystem though. Facebook even wrote a package to generate types automatically for existing code https://github.com/Instagram/MonkeyType .

Re: startups, it's more useful in a large, mature code-base rather than smaller projects still in the design-churn phase.

It’s important to introduce it early enough that it’s still feasible to introduce it. Otherwise the architecture of the code will get in the way—defining classes deep within module hierarchies for objects that get returned up is painful.

Re: Python at Netflix

#217

Post by dorgo »

abritinthebay wrote: post #189
glckr wrote: post #63

Interesting. I had it in my head that Netflix was a pure Java shop. Was I wrong in the first place, or has something changed?

We're everything! JS is heavily used, as is Groovy, Java, Python, C and C ++, etc. Depends on the project and the team. We're not about limiting people, but the choice should be justifiable (like... Brainfuck is probably not ok)

https://en.wikipedia.org/wiki/Brainfuck

Re: Python at Netflix

#218

Post by mixmastamyk »

weberc2 wrote: post #212
jnwatson wrote: post #209

Earlier quoted context omitted.

Both problems have built-in debug solutions in recent versions of python. The event loop will literally print out all the un-awaited coroutines when it exits, and you can enable debug on the event loop and have it print out every time a coroutine takes longer than a configurable amount of time.

> The event loop will literally print out all the un-awaited coroutines when it exits IIRC, I've only ever seen "unawaited coroutine found" (or similar) errors; I've never seen anything that points to a specific unawaited coroutine. In either case, a bug in prod is still many times worse than compile time type error. > you can enable debug on the event loop and have it print out every time a coroutine takes longer th…

These are not really an issue in vfx production and other things Python is used for.

Re: Python at Netflix

#219

Post by gvd »

weberc2 wrote: post #212
jnwatson wrote: post #209

Earlier quoted context omitted.

Both problems have built-in debug solutions in recent versions of python. The event loop will literally print out all the un-awaited coroutines when it exits, and you can enable debug on the event loop and have it print out every time a coroutine takes longer than a configurable amount of time.

> The event loop will literally print out all the un-awaited coroutines when it exits IIRC, I've only ever seen "unawaited coroutine found" (or similar) errors; I've never seen anything that points to a specific unawaited coroutine. In either case, a bug in prod is still many times worse than compile time type error. > you can enable debug on the event loop and have it print out every time a coroutine takes longer th…

"(yet)", but then it's still a dynamic language (no typing, not dynamic as in hip). The reason for python is mainly:

- it's easy to learn - we've go numpy

If I were to pick a language to build significant infrastructure with, I wouldn't choose python.

Re: Python at Netflix

#220

Post by weberc2 »

mixmastamyk wrote: post #218
weberc2 wrote: post #212

Earlier quoted context omitted.

> The event loop will literally print out all the un-awaited coroutines when it exits IIRC, I've only ever seen "unawaited coroutine found" (or similar) errors; I've never seen anything that points to a specific unawaited coroutine. In either case, a bug in prod is still many times worse than compile time type error. > you can enable debug on the event loop and have it print out every time a coroutine takes longer th…

These are not really an issue in vfx production and other things Python is used for.

It’s a problem for lots of things Python is used for, but maybe not vfx (whatever that is).
Post reply on HN