Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Python at Scale: Strict Modules

instagram-engineering.com

Post reply on HN
Print view View original

231–240 of 259 posts

Re: Python at Scale: Strict Modules

#231

Post by timothycrosley »

weberc2 wrote: post #217
timothycrosley wrote: post #57

Earlier quoted context omitted.

I hate the fact that you may be right, because I really don't like Go in many ways: - I hate it's module system and package eco-system story. - I don't like its syntax. - I don't like its error handling. - I'd much prefer gradual typing. - I want to maintain the ability to use interactive interpreters. - I don't like the fact that instead of being community driven it is Google driven. But, anecdotally, I see go being…

These are all fair points. I really enjoy Python, but there are too many things I fight with on a regular basis that simply aren’t issues in Go. It could be so much better if (1) there was a better type system (mypy is unnecessarily shoehorned into the syntax and still very broken—can’t even express recursive types like JSON), (2) a good way to constrain the dynamism so performance could be improved, and (3) a better…

> type-system

I agree, but if you for instance look at the TypeScript comparison sub-thread, you'll see that all the issues with both the syntax and implementation of the type-system are being aggressively resolved, and likely will all be so by 3.9.

> Good way to constrain the dynamism so performance could be improved

Couldn't agree more!

> environment

I find poetry a joy to use. If you want to bypass venvs all together, there's a lot of work to make that a reality, such as https://github.com/David-OConnor/pyflow.

> packaging

Python in 3.5 added complete zip app support, which has improved this dramatically from my perspective. Extended by things like shiv https://github.com/linkedin/shiv make it fairly complete.

> async/await

This is interesting to me. I prefer async/await in general, because it has become a standard across programming languages and I find it really easy to reason about. I also find channels to be too widely seen as a cure-all, when the only study so far has shown they actually led to an increase bug count. But I don't discount the value of real-parallelism, and am glad to see that Python has been pushing harder on that lately, with things like subshells that allow bypassing the GIL on a single thread.

Re: Python at Scale: Strict Modules

#232

Post by andreareina »

k_sze wrote: post #218

Another thing that I would like to see in some kind of strict mode is the ability to mark explicit exports like in JavaScript modules. I often want to import multiple things globally at the top of a module because they are shared by multiple class or function definitions that I am writing. However, such imports end up being exposed to and usable by the consumers of my module, even though the consumers should really h…

3. Import as normal, and leave it to the principle of "we're all consenting adults"; unless something is explicitly called out as being part of the public API I consider Law of Demeter[1] "violation" the same as accessing _var.

[1] https://en.wikipedia.org/wiki/Law_of_Demeter

Re: Python at Scale: Strict Modules

#233

Post by gpderetta »

chubot wrote: post #229
gpderetta wrote: post #205

Earlier quoted context omitted.

> type checking and metaprogramming are fundamentally at odds [1] I would say that dynamic metaprogramming is at odd with type checking (and optimizations, and in general understanding the behaviour of a program statically). But of course metaprogramming can be done perfectly fine in a statically typed language.

There's less conflict if you release the features in the same compiler of course, but it's definitely not "perfectly fine". My blog posts give some more color on that, but also see: https://discuss.ocaml.org/t/the-future-of-ppx/3766 (breakage) https://words.steveklabnik.com/an-overview-of-macros-in-rust (breakage) Also, OCaml has had at least 4 different metaprogramming systems -- ocamlmeta, camlp4, ppx, etc. Rust is…

There isn't anything specific of static type systems in those two links. Both ocaml and rust had powerful plugin systems that exposed the internal of the the compiler AST and they decided that they do not want to expose it as a stable interface. But exposing implementation details is not required to have a powerful metaprogramming environment. As far as I understand rust macros never did expose these internals and had no such issues.

Also compare with C++ metaprogramming (whose syntax is certainly awful, although it has been continually improving), but works perfectly fine with its type system (in fact most metaprogramming in C++ is done via the type system).

On the other hand CPython also exposes the runtime internals to plugins and de facto that prevents the language from evolving and alternative implementations to gain a foothold, so the issue of exposing implementation details preventing language evolution is not restricted to static languages.

edit: the hard part of typing and metaprogramming is making sure your metaprograms are well typed, i.e. the generated program is guaranteed to typecheck. This is great, but it is not a strict requirement, if you are happy with syntax macros a la rust or unconstrained templates a la C++, there is no particular issue. Your genrated program will be still be typechecked at compile time, which is still better than having a runtime error because of a bad metaprogram.

Re: Python at Scale: Strict Modules

#234

Post by timothycrosley »

weberc2 wrote: post #216
timothycrosley wrote: post #155

Earlier quoted context omitted.

I'm curious what you see as the tooling benefits of go?

Build tooling (“go build” vs setup.py), type checker, text editor support (hovering over a symbol for the type and docstring), documentation generator / godoc.org, dependency management (pip is great but it’s not reproducible; go’s toolchain is only modestly better here IMO), no need for virtualenvs, etc. I’m sure I’m missing several.

> build tooling

All my projects now use poetry for the full build tooling and I love it. No setup.py needed just include any settings in the standard pyproject.toml file example: https://github.com/timothycrosley/portray/blob/master/pyproj..., which can be generated with poetry's help using poetry init.

> text-editor support

I feel like Python with type hints (for all their current flaws) does give you this exactly.

> dependency management

Again I think poetry solves the problems here very nicely

> documentation generator

Personally, I like portray better than anything in the Golang world for this https://timothycrosley.github.io/portray/ I may be biased since I wrote it.

I write a lot of Python tools so I'm genuinely curious because if there were unfilled needs I would want to address them as one of my 52 projects: https://timothycrosley.com/

Re: Python at Scale: Strict Modules

#235

Post by Jefro118 »

chenzhekl wrote: post #168
timothycrosley wrote: post #18

More and more I want someone to create a new language that amounts to a strict subset of Python, with mypy built-in, and is compilable into machine code. Python has by far my favorite syntax, community, and in my experience leads to the greatest productivity. There just happens to be a lot of overly dynamic features, that aren't even used by most, but used just enough to hold back optimization and structural improvem…

How about Nim? It has Python-like syntax and is as fast as C. https://nim-lang.org/

Just in case it's of interest to anyone reading this, I interviewed the designer of Nim, Andreas, about his design choices and what he learned from Python and the C family here: https://sourcesort.com/interview/andreas-rumpf-on-creating-a...

Gives some good insight into where Nim is going in the future too.

Re: Python at Scale: Strict Modules

#236

Post by carapace »

rswail wrote: post #179
carapace wrote: post #161

Earlier quoted context omitted.

> I'm not sure what good these kinds of bets are. I'm just putting some numbers on my confidence level. > All code could be made leaner and smaller. What's your point? That Instagram's code could be made much leaner and smaller. I feel like I've been very clear about that.

And from Instagram's POV, the ROI on that would be much less than putting in the sort of belts and braces that the article talked about. They don't have the time or space to engage in a massive technical debt reduction program, they're too busy destroying Snapchat and other competitors, reacting to TikTok, implementing an entirely new IGTV video service that provides their customers (ie advertisers and marketers) the…

Dude, sincerely, thank you. I feel like this is the sane answer I was waiting for. Cheers! (and for your other comment in re: what all Instagram does. I appreciate it.)

You're totally right.

Re: Python at Scale: Strict Modules

#237

Post by carapace »

rswail wrote: post #176
carapace wrote: post #137

Earlier quoted context omitted.

> Instagram does extremely complicated things. Name three? Seriously, you're right, I've never used it. Wow me.

From a user's perspective, Instagram has: a) a way to post pictures/videos/sound recordings to a public feed. The pictures can include overlays of links to other users, to other posts, to song lyrics that play in sync with the music, etc etc. Users viewing their posts get the ability to comment/like/link, with automatic language detection and translation on demand. b) a way to see how other users interact with their…

Cheers! Thank you.

Re: Python at Scale: Strict Modules

#238

Post by carapace »

timothycrosley wrote: post #156
carapace wrote: post #58

Earlier quoted context omitted.

What about RPython? https://rpython.readthedocs.io/en/latest/rpython.html

As far as I understand it, RPython isnt' really meant for actually writing programs in: > Do I have to rewrite my programs in RPython? > No, and you shouldn’t try. First and foremost, RPython is a language designed for writing interpreters. It is a restricted subset of Python. If your program is not an interpreter but tries to do “real things”, like use any part of the standard Python library or any 3rd-party library…

Hmm, still it might make a good foundation for the sort of thing you're talking about, eh?

Cheers.

Re: Python at Scale: Strict Modules

#239

Post by weberc2 »

joshuamorton wrote: post #228
weberc2 wrote: post #215

Earlier quoted context omitted.

The opposite is true in my experience. Most of that boilerplate is brackets and indentation, which visually frame the interesting bits, drawing your eye to them. This is, of course, subjective, but I use both regularly and at worst this is not a problem for Go.

The problem here is that there is boilerplate at all. There shouldn't be. Boilerplate distracts from what is actually going on. I can generally identify code smells from the shape of python code (like, blur all the text so I can't read the words, and the shape of the blocks tells me everything I need), I can't do the same in go, because there's so much more indentation and visual stuff happening, and most of it (boil…

Like I said, I disagree with this. I suspect this is either because you're very experienced with Python and relatively inexperienced with Go, or perhaps you're simply an outlier. I think if you surveyed developers who are very experienced with Python and have at least a few months of experience with Go, you'll find people say that it's easier to identify issues in Go code--and I think this largely comes down to the role the boilerplate has in visually "framing" or "structuring" (i.e., providing "shape" to) the code.

Have a look at Haskell which goes to great lengths to eliminate boilerplate and I think you'll experience the opposite--Haskell becomes very difficult to read precisely because the code is so dense. Similarly, take the indentation, newlines, etc out of a JavaScript file or JSON blob (minify it, more or less) and see if it's more or less readable as a result. I think you'll find that visual structure is actually important.

Re: Python at Scale: Strict Modules

#240

Post by weberc2 »

timothycrosley wrote: post #234
weberc2 wrote: post #216

Earlier quoted context omitted.

Build tooling (“go build” vs setup.py), type checker, text editor support (hovering over a symbol for the type and docstring), documentation generator / godoc.org, dependency management (pip is great but it’s not reproducible; go’s toolchain is only modestly better here IMO), no need for virtualenvs, etc. I’m sure I’m missing several.

> build tooling All my projects now use poetry for the full build tooling and I love it. No setup.py needed just include any settings in the standard pyproject.toml file example: https://github.com/timothycrosley/portray/blob/master/pyproj... , which can be generated with poetry's help using poetry init. > text-editor support I feel like Python with type hints (for all their current flaws) does give you this exactly.…

> All my projects now use poetry for the full build tooling and I love it. No setup.py needed just include any settings in the standard pyproject.toml file example: https://github.com/timothycrosley/portray/blob/master/pyproj..., which can be generated with poetry's help using poetry init.

We have yet to try poetry in our org. I'm hesitant to stray off the well-trodden path, but it might be worth a shot. Any idea about installing packages with system dependencies? Packages like `pygraphviz` (which depends on the `graphviz` or `graphviz-devel` system library) has always given us a lot of trouble, for example.

> I feel like Python with type hints (for all their current flaws) does give you this exactly.

I've noticed that some editors try to use these hints, but they seem to have a hard time in many cases loading the modules. It's possible that the editor extensions (e.g., VS Code) are just buggy, but it's still a problem. Further, they require that all of your dependencies have annotations or type stubs.

> Personally, I like portray better than anything in the Golang world for this https://timothycrosley.github.io/portray/ I may be biased since I wrote it.

I haven't tried portray.

The killer thing about Go's documentation generation is that it uses type annotations and exposes them in the generated documentation. This is critical because 95% of the reason I'm looking at documentation (especially in Python) is because I need to know the type signatures (and often Python docs omit types, or the types are wrong or vague--e.g., "the type is 'binary'" with no indication if that means a bytestring or a BytesIO or what). This is tablestakes for documentation systems in statically typed languages, but I have yet to find a Python tool that does this well. Further, `godoc.org` also generates links to types including across packages--this is _not_ tablestakes for statically typed languages--so you just have to click the type name and it will take you to the docs for other packages. Further, there is no CI needed to build/publish your documentation; `godoc.org` just needs access to your repo on github or elsewhere (you can run your own godoc.org inside your corporate firewall). Another nice-to-have feature is that documentation is just comments; there's no formal/obscure syntax a la sphinx.

> I write a lot of Python tools so I'm genuinely curious because if there were unfilled needs I would want to address them as one of my 52 projects: https://timothycrosley.com/

Cool. I'll take a look!

Post reply on HN