Metaprogramming in Python • Hacker News Board

Re: Metaprogramming in Python

#31

Post by narush » Sat, Sep 17, 2022, 3:33 PM UTC

This is a cool overview, and I certainly learned new things about the Python language from it. Thanks for posting!

We do lots of Python metaprogramming at Mito [1], but generally avoid all of this fancy Python fluff to get it done. Specifically, we avoid metaclasses, invisible decorators, etc. Instead, we take a much simpler approach of having Python code that literally populates as template .py file, and then writes it to the correct location in our codebase.

As a concrete example: we’re a spreadsheet, so we let our users transform data in a bunch of different ways - adding a column, writing a formula, deleting some rows. Anytime I want to add a new transform (say, encoding a column), I tell the metaprogramming package “python -m metaprogramming step —name “Encoding A Column”. It will ask me some questions about the parameters and types of those parameters, and then write most of the 4-6 boilerplate Python and Typescript files I need automatically! You can see it here [2].

This is still metaprogramming (it’s certainly code that writes code). But the code you end up with at the end of the day is very simple Python code that is extremely easy to understand / maintain long-term.

I’ll pass on the fancy stuff for now. Thanks though!

[1] https://trymito.io

[2] https://github.com/mito-ds/monorepo/blob/dev/mitosheet/dev/c...

Re: Metaprogramming in Python

#32

Post by wheelerof4te » Sat, Sep 17, 2022, 3:38 PM UTC

kofejnik wrote: post #21

wheelerof4te wrote: post #6

It is shocking how much vodoo magic there is in a programming language marketed as "easy to learn and use". And with the recent addition of new syntax, I would hardly call Python easy.

Ran into this face-first, just yesterday. I'm converting a rather old, rather big API (~400 endpoints) from Python 2.7 into 3, and apparently in Python 2, _hasattr(something, someattribute)_ just returns False if attribute access throws an exception! Specifically, if you have class A(): @property def a(self): raise Exception('go away') a = A() then hasattr(a, 'a') will return False in Python 2.7 (and throw as expecte…

This is a rather unusual problem and I would call this design flawed. Defining useless methods that only exist to raise an exception is in my opinion a waste of space, both virtual and textual.

Re: Metaprogramming in Python

#33

Post by zelphirkalt » Sat, Sep 17, 2022, 3:39 PM UTC

behnamoh wrote: post #17

It’s interesting that meta programming, one of the main selling points of lisps, is actually pretty common and possible in other languages as well. Lisp’s “code = data” achieves meta programming in a straightforward way, but this link shows that it’s not necessary. In fact, Python’s way might even be better because the language sorta “gets outta the way” because it’s really darn simple.

The article however, does not show anything comparable to lispy metaprogramming.

I do not see a single new keyword being defined. I do not see anything changing the order of evaluation. Only to give 2 examples.

Recently I implemented a new kind of "define" in a Scheme, which allows for specifying contracts for a function. I don't think such a thing is possible using the tools shown in the article. As such, the things shown in the article are not really that meta, but rather parts of the Python language already. It is not like it is adding anything to Python, which was not there. It is just usage of some concepts Python already has. This is conceptually different from transforming source code to other source code, which then creates a new concept in the language itself. It is not like I can mold Python into whatever I want, but rather stay in the corset of the language facilities. Which of course is not as nice as lispy languages with good macro system, when it gets to creating DSLs and other conveniences.

So overall I am not so blown away by the article.

Re: Metaprogramming in Python

#34

Post by kofejnik » Sat, Sep 17, 2022, 3:47 PM UTC

wheelerof4te wrote: post #32

kofejnik wrote: post #21

Earlier quoted context omitted.

Ran into this face-first, just yesterday. I'm converting a rather old, rather big API (~400 endpoints) from Python 2.7 into 3, and apparently in Python 2, _hasattr(something, someattribute)_ just returns False if attribute access throws an exception! Specifically, if you have class A(): @property def a(self): raise Exception('go away') a = A() then hasattr(a, 'a') will return False in Python 2.7 (and throw as expecte…

This is a rather unusual problem and I would call this design flawed. Defining useless methods that only exist to raise an exception is in my opinion a waste of space, both virtual and textual.

obv, this is just a minimal demo (i don't name my classes A!); in actual codebase, there's a complicated calculation that throws up under some circumstances

Re: Metaprogramming in Python

#35

Post by ok123456 » Sat, Sep 17, 2022, 3:53 PM UTC

It's interesting how differently two similar dynlangs, Python and Ruby, decided to offer meta-programming. Ruby decided to offer a very simple way of evaluating code at construction, but python has you define custom metaclasses. One sees it as a routine way of extending the language, and the other treats it as a 'black-art'.

Metaprogramming done wrong, no matter the mechanism, is very painful. So, I'm not sure which way is better.

Re: Metaprogramming in Python

#36

Post by BerislavLopac » Sat, Sep 17, 2022, 3:57 PM UTC

narush wrote: post #31

This is a cool overview, and I certainly learned new things about the Python language from it. Thanks for posting! We do lots of Python metaprogramming at Mito [1], but generally avoid all of this fancy Python fluff to get it done. Specifically, we avoid metaclasses, invisible decorators, etc. Instead, we take a much simpler approach of having Python code that literally populates as template .py file, and then writes…

I personally consider code generation an anti-pattern in Python. With its dynamic nature and the two-step execution model, Python is essentially its own macro language, and any generation can be done at runtime.

Re: Metaprogramming in Python

#37

Post by kstrauser » Sat, Sep 17, 2022, 4:16 PM UTC

wheelerof4te wrote: post #6

It is shocking how much vodoo magic there is in a programming language marketed as "easy to learn and use". And with the recent addition of new syntax, I would hardly call Python easy.

I’ve been writing Python professionally for over 20 years, and I’ve needed to use the things here once. It was highly encapsulated inside one module of an ORM-type project, where other code wanted to use it without having to know anything about its implementation details.

You can do these things in the extremely rare cases that you must, but other than that you shouldn’t.

Re: Metaprogramming in Python

#38

Post by pydry » Sat, Sep 17, 2022, 4:39 PM UTC

samwillis wrote: post #22

This oft-repeated quote is always relevant on Python metaclass posts: “[Metaclasses] are deeper magic than 99% of users should ever worry about. If you wonder whether you need them, you don’t (the people who actually need them know with certainty that they need them, and don’t need an explanation about why).” Tim Peters, Inventor of the timsort algorithm and prolific Python contributor https://www.oreilly.com/library…

I've run across people in three different companies who decided to use them for no good reason at all.

I was only around for one of them while he did it but it seemed pretty clear that he was insecure and trying to prove that he was a more experienced programmer by using the most advanced features.

I call it "the advanced beginner metaclass trap".

Re: Metaprogramming in Python

#39

Post by stuckinhell » Sat, Sep 17, 2022, 4:49 PM UTC

As someone who just dabbles in python for small scripts, this is fascinating to me.

You can use some metaprogramming to create very clean interface points in python! I always wondered how django did so much with very clean readable implementations for end users.

Re: Metaprogramming in Python

#40

Post by stuckinhell » Sat, Sep 17, 2022, 5:24 PM UTC

carapace wrote: post #24

This is awesome! Now, never ever do it. Python is powerful and flexible enough that you don't need metaprogramming. (I mean this quite literally: I sincerely doubt that there any code in Python using metaclasses etc., that wouldn't be more clear and maintainable if rewritten in "plain old" Python without them.) (With the caveat that I'm not including "art" projects, I'm talking about working production code.) (In cas…

The example shows that django models uses metaclasses heavily. So I'm assuming if you needed to extend django models, you'd have to do it via metaclasses.