Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Python at Scale: Strict Modules

instagram-engineering.com

Post reply on HN
Print view View original

141–150 of 259 posts

Re: Python at Scale: Strict Modules

#141

Post by carapace »

scrollaway wrote: post #98
carapace wrote: post #48

Earlier quoted context omitted.

No, I'm seriously you guys. Python is too expressive to require mega-LoC for that site. You could implement an OS, relational DB, spreadsheet, and optimizing compiler all in less than that.

As orf said you have no idea about their codebase. And you have no idea what's included in that statement -- given that they talk about startup time, they most likely are taking into account the whole framework, a plethora of admin and analytics tools, lots of debugging / debug-only infrastructure, migrations, lots of tooling whose sole purpose is making it easier to work in large teams, etc… (And for the record, Lin…

> As orf said you have no idea about their codebase.

I do too: It's Python and it's several million lines.

Metaphor: you've got three pallets of goods and have hired three trucks to move them. I don't have to know how you wrapped the pallets to know that you brought two too many trucks.

I don't have to know the details of what's included in "Instagram Server" et. al. to make this call (obviously) based on my experience and first-hand knowledge of similar codebases. Frankly, I am kind of disappointed in the pushback I'm getting on this. The only reason to have a multi-million line Python project is for the entertainment of devs, or, worse yet, job security.

Let me put it this way, if the CTO of Instagram showed up here I would be willing to bet US$100,000 that I could reduce the Instagram code by 90% in six months. (Do you think the devs there would appreciate that? Even the one that got laid off as a result?)

If I sound cynical it's only because I've seen this sort of thing for myself. I'm not trying to say that the Instagram devs are dumb or nefarious, this kind of code happens organically and often despite our best efforts. But that code needs a diet. I'm sure of that.

- - - -

edit: In re:

> (And for the record, Linux is ~37 million lines of actual code, Postgres ~2 million, and gcc ~8 million)

So, call it 50M LoC, what's your ratio for Python/C? Meaning, how many lines of C code are replaced, on average, by one line of Python?

And how feature-complete are we talking? POSIX? GCC targets a lot of languages and platforms, eh?

If you were going for an integrated system, like Oberon OS or a Smalltalk IDE, I think my claim is still plausible, eh?

:)

Re: Python at Scale: Strict Modules

#142

Post by timothycrosley »

staticassertion wrote: post #136
ledauphin wrote: post #61

Earlier quoted context omitted.

yeah, I'm excited about this approach. but it's a long way from being a realistic approach for folks outside Dropbox, it seems.

It's a long way from being a realistic approach for anyone, including Dropbox, really.

What would you say are the biggest blockers to becoming realistic? I saw on the README they need tools in the Python ecosystem to start utilizing them, which I can help with starting with isort, beyond that I'd want to do whatever I can to help the project succeed.

Re: Python at Scale: Strict Modules

#143

Post by scrollaway »

carapace wrote: post #141
scrollaway wrote: post #98

Earlier quoted context omitted.

As orf said you have no idea about their codebase. And you have no idea what's included in that statement -- given that they talk about startup time, they most likely are taking into account the whole framework, a plethora of admin and analytics tools, lots of debugging / debug-only infrastructure, migrations, lots of tooling whose sole purpose is making it easier to work in large teams, etc… (And for the record, Lin…

> As orf said you have no idea about their codebase. I do too: It's Python and it's several million lines. Metaphor: you've got three pallets of goods and have hired three trucks to move them. I don't have to know how you wrapped the pallets to know that you brought two too many trucks. I don't have to know the details of what's included in "Instagram Server" et. al. to make this call (obviously) based on my experien…

I'm not sure what good these kinds of bets are. All code could be made leaner and smaller. What's your point?

Re: Python at Scale: Strict Modules

#144

Post by clintonb »

cpeterso wrote: post #100
timothycrosley wrote: post #18

More and more I want someone to create a new language that amounts to a strict subset of Python, with mypy built-in, and is compilable into machine code. Python has by far my favorite syntax, community, and in my experience leads to the greatest productivity. There just happens to be a lot of overly dynamic features, that aren't even used by most, but used just enough to hold back optimization and structural improvem…

I like Python, but I often wonder how many developers use Python because they actually use dynamic language features versus just liking the languages' clean syntax and library ecosystem. I'm surprised languages that offer both REPL (for development) and AOT native compilation (for production), like OCaml, are not more popular. Evidence that syntax matters, I guess. :) mypy and mypyc are interesting but their compile-…

Don’t underestimate inertia. I’ve worked with Python and Django for seven years. I know the libraries in the ecosystem. I know the framework. It’s far easier for me to start a project with Django than to learn another framework or language.

Re: Python at Scale: Strict Modules

#146

Post by cutler »

dunefox wrote: post #116
coldtea wrote: post #71

Earlier quoted context omitted.

Countless companies, huge and small -- from Apple and Amazon, to Google and your friendly local startup, plus all the enterprise world that's not a .NET shop... In what parallel universe is not Java immensely popular or not used for green projects?

Hopefully in every universe where Kotlin exists.

Kotlin != Java .... unfortunately. Sorry Jetbrains.

Re: Python at Scale: Strict Modules

#148

Post by paulddraper »

ashishb wrote: post #122
paulddraper wrote: post #117

Earlier quoted context omitted.

TypeScript is perfectly this. (And other gradually typed solutions; TS is simply the most popular one.) You have the madness of thousand of developers flinging code at the universe due to the easiness of browsers, JS, and npm. This results in great speed, but not great quality. When your project/company now wants quality, you keep your code but transition to types. (In OSS space, Angular and Yarn projects have both d…

Afaik, typescript is pretty bad in terms of catching some basic errors. Types are not enforced. A caller can change sync function to async, breaking the functionality downstream.

Yes gradually typed languages are usually looser/more flexible about static typing.

> A caller can change sync function to async, breaking the functionality downstream.

I think you mean callee?

Are you taking about using a returned result? Because most languages permit ignoring return values.

If you want to check out promise use, check out https://tsetse.info/must-use-promises

Re: Python at Scale: Strict Modules

#149

Post by joshuamorton »

DonaldPShimoda wrote: post #103
miki123211 wrote: post #21

This is yet another example of the divide between wizarding and engineering[1]. When you're a small startup, what matters is the expressiveness of your language, and the ability do do a lot of things very very quickly. Type safety, performance, readability, those things don't matter. You're just a bunch of engineers who know the whole codebase inside out, you're pretty certain of what you're doing. In short, you're w…

> When you're a small startup, what matters is the expressiveness of your language, and the ability do do a lot of things very very quickly. Type safety, performance, readability, those things don't matter. You're just a bunch of engineers who know the whole codebase inside out, you're pretty certain of what you're doing. I'm not familiar with this use of the term "expressiveness". My understanding is that expressive…

I expect in this context, expressiveness means something like "the ability to describe the relevant stuff in the code with minimal noise", which might map to having good abstraction.

I find that pythons OOP + functional aspects, combined with a good understanding of the language hits a sweet spot here. One that simply can't be reached in c/cpp/go/java/haskell, and which is much easier to reach than in js/rust/other langs where I think it is possible.

Re: Python at Scale: Strict Modules

#150

Post by kbd »

nothrabannosir wrote: post #51
kbd wrote: post #8

Earlier quoted context omitted.

This has nothing to do with types. It's more about static guarantees the language gives about module import behavior.

In OP's defence: > So that's a third pain point for us. Mutable global state is not merely available in Python, it's underfoot everywhere you look: every module, every class, every list or dictionary or set attached to a module or class, every singleton object created at module level. It requires discipline and some Python expertise to avoid accidentally polluting global state at runtime of your program. > One reason…

How is "our developers really like Python even on a million-line codebase, despite its global mutable state requiring discipline" "quite damning"?
Post reply on HN