Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Best Practices for Working with Configuration in Python Applications

tech.preferred.jp

Post reply on HN
Print view View original

31–40 of 69 posts

Re: Best Practices for Working with Configuration in Python Applications

#31

Post by anentropic »

Shout out here for pydantic BaseSettings https://pydantic-docs.helpmanual.io/usage/settings/

That provides typed and validated auto-loading from env vars. I have been quite happy with that in conjunction with an optional .toml file, to do flexible config cleanly and simply like:

    import toml

    from myproj.conf.types import Settings  # a pydantic BaseSettings model


    try:
        _config = toml.load('myproj.toml')
    except FileNotFoundError:
        _config = {}


    settings = Settings(
        **{key.upper(): val for key, val in _config.items()}
    )

Re: Best Practices for Working with Configuration in Python Applications

#32

Post by kingosticks »

frumiousirc wrote: post #26
kingosticks wrote: post #9

Earlier quoted context omitted.

Isn't that basically the same end result as using json.loads except a different format (that has no actual spec).

JSON does not support comments nor string interpolation. Python ConfigParser language does.

True, you do get string interpolation but the comment support in ConfigParser isn't very good. Although actually they may have fixed some of that in Python 3 but I'm still using workarounds.

To be clear, I am not suggesting using JSON for config, I think that would be my last choice. My point is that ConfigParser isn't really an alternative to rolling your own if you want decent validation etc (those spec files are horrible to use). You very quickly need to start extending ConfigParser to the point where you've started rolling your own. And at that point you'd be better off with one of the other (tested) solutions already suggested.

Re: Best Practices for Working with Configuration in Python Applications

#33

Post by thomk »

Unless the end user is not technical, use a .py file and force them to subclass your Configuration class which has an __init_subclass__ method so you can enforce rules.

When you are ready to move to a more generic solution, your .config or .yml file can generate these.

The advantage here is both flexibility (it's Python) and control (allow/disallow whatever you want).

If you need nested items, use nested classes.

Re: Best Practices for Working with Configuration in Python Applications

#34

Post by ralston3 »

thomk wrote: post #33

Unless the end user is not technical, use a .py file and force them to subclass your Configuration class which has an __init_subclass__ method so you can enforce rules. When you are ready to move to a more generic solution, your .config or .yml file can generate these. The advantage here is both flexibility (it's Python) and control (allow/disallow whatever you want). If you need nested items, use nested classes.

This.

Until you the app reaches the level of advanced yaml config files for cloud deployments, it’s really hard to beat a “config.py” that does a single read of all your ENV_VARS at startup

Re: Best Practices for Working with Configuration in Python Applications

#35

Post by madman_bob »

albi_lander wrote: post #29
ungawatkt wrote: post #14

Earlier quoted context omitted.

Marshmallow. You can use its schema validation for any dict/json, which makes it a nice fit for validating json config files (which mitigates some of the json concerns from the article). Just immediately move the json.reads through a schema validate, build some classes around it for different config files. marshmallow.readthedocs.io/en/

dataclass_json is also very useful for schema validation. It combines python's native dataclass objects with marshmallow's schema to provide additional functionalities simply through a @dataclass_json decorator on your dataclass. https://lidatong.github.io/dataclasses-json/

I made the similar dataclasses_serialization library. It doesn't require a special decorator on your classes, and is extensible for custom classes, and custom serialization methods (JSON and BSON provided by default).

https://github.com/madman-bob/python-dataclasses-serializati...

Re: Best Practices for Working with Configuration in Python Applications

#36

Post by markus23 »

As already written by others, the article does not go very deep and is missing many essentials. What I was mostly missing is more about keeping configuration parameters as simple as possible. A much more detailed best practices can be found here: https://www.libelektra.org/ftp/elektra/slides/cm/

Re: Best Practices for Working with Configuration in Python Applications

#37

Post by markus23 »

fermigier wrote: post #11

Quick list of Python libraries that help with application configuration: - Python application configuration -> https://github.com/edaniszewski/bison - Configuration with env variables for Python -> https://github.com/hynek/environ_config - Configuration library for python projects -> https://github.com/willkg/everett - Strict separation of config from code -> https://github.com/henriquebastos/python-decouple This is…

https://www.libelektra.org/tutorials/python-bindings

Re: Best Practices for Working with Configuration in Python Applications

#38

Post by brewmarche »

aucontraire wrote: post #5
alanfranz wrote: post #4

Earlier quoted context omitted.

Your question implies that you don't know about the nuisances of the datetime library :-) (see https://docs.python.org/3/library/datetime.html it's the first paragraph!) Python datetime objects, by design, can be naive or timezone-aware. Timezone-aware datetime objects are OK; they identify a certain instant in time. Naive datetime objects are Python-only abstractions (AFAIK) that don't identify anything in the real…

I agree that python datetime objects are problematic, but for the opposite reason. It is tzinfo that is the sneaky disaster, the plain datetimes are fine. Transparent timezone awareness always fail, unless you are 100% certain that a tzaware datetime object will remain uncoverted from the very top to the very bottom of the stack and all the way up again no matter who is reading and what they are doing. For longterm m…

Actually UTC + timezone is exactly the wrong thing for "wall clock times" (things like meetings or departures where the time at the location is relevant).

The conversion to UTC will lose the original local time so you cannot retrieve it once time zone data changes, unless you perform reconversions every time you detect such a change in tzdata. And countries changing time zones happens more often than we think (and also on short notice).

Thus it is important to distinguish between instants (e.g. for recording when exactly something happened after the fact) and wall clock time (e.g. for coordinating people and goods at a certain place, like meetings, concerts, departure times). For the former use UTC, for the latter use a localised time zone (e.g. Europe/Rome), not an offset time zone (e.g. not +0200).

For more information Jon Skeet has written about this multiple times.

Re: Best Practices for Working with Configuration in Python Applications

#39

Post by macintux »

fermigier wrote: post #11

Quick list of Python libraries that help with application configuration: - Python application configuration -> https://github.com/edaniszewski/bison - Configuration with env variables for Python -> https://github.com/hynek/environ_config - Configuration library for python projects -> https://github.com/willkg/everett - Strict separation of config from code -> https://github.com/henriquebastos/python-decouple This is…

Cerberus for schema validation.

https://docs.python-cerberus.org/en/stable/

Re: Best Practices for Working with Configuration in Python Applications

#40

Post by kyawzazaw »

andyljones wrote: post #12
fermigier wrote: post #11

Quick list of Python libraries that help with application configuration: - Python application configuration -> https://github.com/edaniszewski/bison - Configuration with env variables for Python -> https://github.com/hynek/environ_config - Configuration library for python projects -> https://github.com/willkg/everett - Strict separation of config from code -> https://github.com/henriquebastos/python-decouple This is…

The two big ones in machine learning at least are Google's Gin: https://github.com/google/gin-config and Facebook's Hydra: https://hydra.cc/

the name usage of "gin" is quite common, it seems:

https://github.com/gin-gonic/gin

Post reply on HN