Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Python at Netflix

medium.com

Post reply on HN
Print view View original

231–240 of 250 posts

Re: Python at Netflix

#231

Post by aaronblohowiak »

sandGorgon wrote: post #229
aaronblohowiak wrote: post #225

Earlier quoted context omitted.

Our python code does not handle user requests directly. Our team uses python to control the control plane of traffic — we flip dns records, control cross-region proxying, re-steer cdn-based reverse proxying and scaling of the hundreds of micro services that power the Netflix experience. We’ve improvised various custom traffic distribution patterns, operated outside of our normal traffic-shifting workflows, and writte…

hi - so im actually curious about the infrastructure that allows you to connect bpython to it. Is it a rq process that you connect to, etc ? I would love to setup an architecture like this that lets me connect to stuff through a CLI. Also, I'm assuming you are on kubernetes (or something). How does this bpython business work through all those layers ?

We ssh into a box in production, run bpython and import the libraries ( some environmental stuff, boto3, our own code, etc). At that point, there isn’t much difference from a bpython session and the regular application code... Ssh’ing into a production box does involve connecting through a bastion, and knowing which box you want to connect to, but that isn’t too hard with spinnaker.

Re: Python at Netflix

#232

Post by badpun »

markbnj wrote: post #230
badpun wrote: post #205

Earlier quoted context omitted.

> A system can be distributed without concurrency. A concurrent system need not be distributed. The only justification for such design I can think of is HA. I.e. you have very light workload that does not saturate a single machine, but nonetheless you create a distributed system with two machines for HA. Such light workload is probably not often the case at Netflix.

> The only justification for such design I can think of is HA. I'm not trying to justify any particular design. I'm just saying that concurrency and distributed processing are different things. And I'm no expert, but I continue to be confused as to what saturation, whether of cpu or i/o - you don't specify - has to do with concurrency? If I have for example n http servers all running one thread and handling enough tr…

> f I have for example n http servers all running one thread and handling enough traffic that they are all at 100 percent CPU that is by some minimal definition a distributed system, but none of the processes in it are concurrent.

Concurrency is broader than just threads in a process. In your example, the whole problem is concurrent.

Re: Python at Netflix

#233

Post by sandGorgon »

aaronblohowiak wrote: post #231
sandGorgon wrote: post #229

Earlier quoted context omitted.

hi - so im actually curious about the infrastructure that allows you to connect bpython to it. Is it a rq process that you connect to, etc ? I would love to setup an architecture like this that lets me connect to stuff through a CLI. Also, I'm assuming you are on kubernetes (or something). How does this bpython business work through all those layers ?

We ssh into a box in production, run bpython and import the libraries ( some environmental stuff, boto3, our own code, etc). At that point, there isn’t much difference from a bpython session and the regular application code... Ssh’ing into a production box does involve connecting through a bastion, and knowing which box you want to connect to, but that isn’t too hard with spinnaker.

pretty cool. We have struggled with doing this in Flask, because the whole codebase starts getting peppered with flask_context everywhere. So importing some libraries and code starts spawning flask.

Not sure if you use different frameworks that dont have this issue.

Re: Python at Netflix

#234

Post by ohithereyou »

gonehome wrote: post #214
ohithereyou wrote: post #198

Earlier quoted context omitted.

The FAANGs support diversity. You can go to any of MIT, Stanford, CalTech, or Harvard.

I know this was just an off the cuff response, but it's a little more nuanced than that. They don't really care where you went or even if you have a degree, but there are so many people applying through the front door that that ends up being a filter everyone uses. There are obviously excellent people outside of that, but it's harder to find them through the noise. This is another reason I think Lamda School is great…

It's partially an off the cuff response, but like all good off the cuff responses, it's meant to cause the reader to think a little bit deeper about the message. I'm glad that you didn't find it beneath you to respond to it as it was not intended to be dismissive of response, nor was it thoughtless.

Many of the FAANGs have pushed for diversity in hiring. I don't have a problem with that. I think an ideal workplace would hire people regardless of their circumstances of birth and lived experience as long as they are qualified to do the job and they, in fact, do a good job there. That's actually the crux of my problem.

When companies have pushed to remove names, genders, and races from resumes before the hiring process because they find that people are treated differently in hiring due to unconscious bias in that process then it smacks of hypocrisy to then blanket filter incoming resumes unless you went to a short list of approved schools or have a buddy on the inside juicing your chances. That sounds like the complete opposite of trying to be diverse in recruitment. I don't think that Lambda School is that great of a response to this - it just adds one more "acceptable" school to the list.

In fact, to a cynic, it sounds like virtue signalling of the highest order to please both Wall Street and the public.

Re: Python at Netflix

#235

Post by luord »

Too wrote: post #228
kbumsik wrote: post #89

Earlier quoted context omitted.

> Python is in fact a type safe language for a while now I like the idea of type annotators but they do not make it a type safe language because types are not checked and they are more like comments. Python will remain a dynamically typed language, and the authors have no desire to ever make type hints mandatory, even by convention. [1] [1]: https://www.python.org/dev/peps/pep-0484/#non-goals

-Wall -Werror is also optional in GCC... Not that i like it but just to show that a lot of languages require a properly set up configuration and CI-infrastructure to make them safe and useful. mypy should be considered equal to -Wall -Werror and anyone who doesn't run it is frankly a bloody idiot.

Your entire reason for calling people not doing what you do "bloody idiots" stems from your opinion that static typing is superior.

It isn't, you're just contributing to a flamewar that has been going on since computers have been a thing.

Re: Python at Netflix

#236

Post by luord »

bouncing wrote: post #9
1ba9115454 wrote: post #2

Most of these use cases would be better served with a type safe language.

I'm going to start a drinking game every time someone says that.

It's been going on for sixty years (something the defenders of both paradigms often seem to ignore) and will never end, you'll die.

Re: Python at Netflix

#237

Post by mixmastamyk »

weberc2 wrote: post #223
mixmastamyk wrote: post #221

Earlier quoted context omitted.

They are using it for things it’s good at, for others they use java. So this subthread is largely a waste of time.

Who is “they”? What is your point?

They is Netflix, and other post-production oriented users. You know, what this article and discussion is about?

Re: Python at Netflix

#238

Post by markbnj »

badpun wrote: post #232
markbnj wrote: post #230

Earlier quoted context omitted.

> The only justification for such design I can think of is HA. I'm not trying to justify any particular design. I'm just saying that concurrency and distributed processing are different things. And I'm no expert, but I continue to be confused as to what saturation, whether of cpu or i/o - you don't specify - has to do with concurrency? If I have for example n http servers all running one thread and handling enough tr…

> f I have for example n http servers all running one thread and handling enough traffic that they are all at 100 percent CPU that is by some minimal definition a distributed system, but none of the processes in it are concurrent. Concurrency is broader than just threads in a process. In your example, the whole problem is concurrent.

Yes of course you're correct. Any system of that kind is concurrent across its architecture. But the current context being considered is that of concurrency as a programming paradigm, which was established by the OP's observations on python's lack of concurrency which sparked the exchange. In the comment you quoted above the statement "none of the processes are concurrent" was meant to make the context clearer.

Re: Python at Netflix

#239

Post by gonehome »

ohithereyou wrote: post #234
gonehome wrote: post #214

Earlier quoted context omitted.

I know this was just an off the cuff response, but it's a little more nuanced than that. They don't really care where you went or even if you have a degree, but there are so many people applying through the front door that that ends up being a filter everyone uses. There are obviously excellent people outside of that, but it's harder to find them through the noise. This is another reason I think Lamda School is great…

It's partially an off the cuff response, but like all good off the cuff responses, it's meant to cause the reader to think a little bit deeper about the message. I'm glad that you didn't find it beneath you to respond to it as it was not intended to be dismissive of response, nor was it thoughtless. Many of the FAANGs have pushed for diversity in hiring. I don't have a problem with that. I think an ideal workplace wo…

Their goal is to hire the best candidates as quickly as possible and given the amount of applications it makes sense for them to focus attention somewhere.

Currently highly selective schools and other famous companies are the easiest filter. Then they don’t want to have unconscious bias from that point on - I don’t think it’s virtue signaling, it’s more just pragmatism.

The reason I think Lambda School is cool is that they're actually focused on the first piece of this, scaling up to give opportunity to capable people the current system ignores and eventually leveraging their reputation to get people interviews. Right now it’s extremely unlikely to get accepted into MIT or Stanford, but Lambda School is incentivized to not do this - if you’re capable of the work they want to be able to scale to admit you.

Re: Python at Netflix

#240

Post by weberc2 »

mixmastamyk wrote: post #237
weberc2 wrote: post #223

Earlier quoted context omitted.

Who is “they”? What is your point?

They is Netflix, and other post-production oriented users. You know, what this article and discussion is about?

It's not at all obvious that "they" refers to "netflix and other post-production oriented users", and your argument is a tautology "Python is good at the things that Python is good at". Obviously. The rest of us are debating what those things are or are not.
Post reply on HN