Best Practices for Working with Configuration in Python Applications
tech.preferred.jp
Best Practices for Working with Configuration in Python Applications
1–10 of 69 posts
Re: Best Practices for Working with Configuration in Python Applications
#2While most points are valid, I feel some pieces are missing in this article.
What should I finally do? How to put everything together without creating an hard-to-maintain mess of casting/parsing/configuration? Should I manually cast strings to integers (or other types) for all and each value I parse? Where do I keep default values? It's cumbersome to have them embedded in code for get().
I usually want a) a default configuration kept in a file b) a way to override that config with other files, but only for certain parts (I don't want to rewrite the configuration every time), c) a way to override the config at launch time (e.g. from cmdline)
The fact that Python is dynamically typed/type hinted only makes it harder than statically typed languages at configuration, where most configuration libraries instantiate an adequate type conversion function to put a string somewhere.
I ultimately found that the latest solution (parse from json) is good enough for most use cases for points a) and b); since json is typed, a decent conversion can happen for 95% of the use cases (for the others, just use a string and manually parse).
A sidenote to the author if he/she's reading: datetime.date objects, just as python's own naive datetime objects, are dangerous objects that can lead to unpredictable results when used with actual time-handling code. I wouldn't use them anywhere in my Python code.
Re: Best Practices for Working with Configuration in Python Applications
#3While most points are valid, I feel some pieces are missing in this article. What should I finally do? How to put everything together without creating an hard-to-maintain mess of casting/parsing/configuration? Should I manually cast strings to integers (or other types) for all and each value I parse? Where do I keep default values? It's cumbersome to have them embedded in code for get(). I usually want a) a default c…
What’s naive and dangerous about Python’s Datetime objects?
Re: Best Practices for Working with Configuration in Python Applications
#4While most points are valid, I feel some pieces are missing in this article. What should I finally do? How to put everything together without creating an hard-to-maintain mess of casting/parsing/configuration? Should I manually cast strings to integers (or other types) for all and each value I parse? Where do I keep default values? It's cumbersome to have them embedded in code for get(). I usually want a) a default c…
What’s naive and dangerous about Python’s Datetime objects?
Your question implies that you don't know about the nuisances of the datetime library :-) (see https://docs.python.org/3/library/datetime.html it's the first paragraph!)
Python datetime objects, by design, can be naive or timezone-aware. Timezone-aware datetime objects are OK; they identify a certain instant in time.
Naive datetime objects are Python-only abstractions (AFAIK) that don't identify anything in the real world; they're highly error prone, because there's no "right" way to use them.
They sort of work properly only if used in a very limited scope - e.g. your own code only, for small sections - but they're risky because they're not a different type (with respect to tz-aware), and it's hard to tell what any code accepting a datetime does if passed a naive object. Some libraries like java.time DO have a similar concept (e.g. LocalTime, LocalDate) but they keep it well separate from the "real" concept (e.g. Instant or Date in Java) so you can't use them accidentally.
Example: you pass a naive datetime object to any library which must translate it to an instant, like an ISO string with a well defined timezone. What does the library do? Throw an exception? Associate an arbitrary timezone (e.g. UTC)? Associate the local, current timezone? There's no "correct" behaviour.
Re: Best Practices for Working with Configuration in Python Applications
#5Earlier quoted context omitted.
What’s naive and dangerous about Python’s Datetime objects?
Your question implies that you don't know about the nuisances of the datetime library :-) (see https://docs.python.org/3/library/datetime.html it's the first paragraph!) Python datetime objects, by design, can be naive or timezone-aware. Timezone-aware datetime objects are OK; they identify a certain instant in time. Naive datetime objects are Python-only abstractions (AFAIK) that don't identify anything in the real…
I agree that python datetime objects are problematic, but for the opposite reason. It is tzinfo that is the sneaky disaster, the plain datetimes are fine.
Transparent timezone awareness always fail, unless you are 100% certain that a tzaware datetime object will remain uncoverted from the very top to the very bottom of the stack and all the way up again no matter who is reading and what they are doing.
For longterm minimization of pain, bugs and effort, you convert datetimes to UTC as early as possible and take them back to some localized version as late as possible (in the frontend, for a webapp, so that the backend never needs to know there is such a thing as timezones (except for separate validation and correction routines, since timezone definitions always end up being incorrect to some degree when you use them at scale)).
If the localization of the datetime is an essential aspect (such as the departure time of a ship leaving port), you store a UTC value together with a record of the location. Only at the latest possible moment of processing, should you do a lookup on the location data to make a local time.
Obviously, there will be exceptions to this rule. If you batch process billions of timestamps under a tight deadline and must do calculations in local time, it might make sense to have the values persisted localized.
Re: Best Practices for Working with Configuration in Python Applications
#6I think argparse covers most of the points mentioned as desirable in this article.
* validate at start (using the type keyword argument for add_argument) * access by name as identifier, not string
And what is more is, that default values are stored with the configuration, plus you add a help text for telling everyone what the argument is for.
One downside is that you command line call gets longer.
Re: Best Practices for Working with Configuration in Python Applications
#7Pretty good guidelines. I like the idea of having a configuration close to the class (perhaps module-level depending on the project size) that uses it. With dataclass the class definition is fairly clean. In addition to that, I'd consider using https://docs.python.org/3/library/dataclasses.html#post-init... for business-specific validations.
Re: Best Practices for Working with Configuration in Python Applications
#8it's worth looking at python's own configparser[1] before rolling your own.
Re: Best Practices for Working with Configuration in Python Applications
#9it's worth looking at python's own configparser[1] before rolling your own. [1] https://docs.python.org/3/library/configparser.html
Isn't that basically the same end result as using json.loads except a different format (that has no actual spec).
Re: Best Practices for Working with Configuration in Python Applications
#10Earlier quoted context omitted.
Your question implies that you don't know about the nuisances of the datetime library :-) (see https://docs.python.org/3/library/datetime.html it's the first paragraph!) Python datetime objects, by design, can be naive or timezone-aware. Timezone-aware datetime objects are OK; they identify a certain instant in time. Naive datetime objects are Python-only abstractions (AFAIK) that don't identify anything in the real…
I agree that python datetime objects are problematic, but for the opposite reason. It is tzinfo that is the sneaky disaster, the plain datetimes are fine. Transparent timezone awareness always fail, unless you are 100% certain that a tzaware datetime object will remain uncoverted from the very top to the very bottom of the stack and all the way up again no matter who is reading and what they are doing. For longterm m…
> I agree that python datetime objects are problematic, but for the opposite reason. It is tzinfo that is the sneaky disaster, the plain datetimes are fine.
Why naive datetimes should be fine? How are they fine? What do they represent?
> For longterm minimization of pain, bugs and effort, you convert datetimes to UTC
This COULD work if naive objects had an IMPLIED UTC in their contract - e.g. naive objects are declared as ALWAYS UTC. Your argument fails as soon as you pass a naive datetime object somewhere in a library/framework and it gets accepted, and/or you try serializing it without augmenting with a TZ. As I said, naive datetime only works if you control 100% of their usage. No libraries, no external points of contact. And, the reason because tz-aware objects sometimes fail for the opposite reason (e.g. libraries assume naive objects) is a fault of the API design (they're not distinct types), but the problem lies in the existence of the naive version, not vice-versa.
For the records: in the backend I always use tz-aware datetime objects with a fixed UTC timezone. That's the best way IMHO not to get crazy with time problems in Python. So, your points about datetime handling are all valid and correct (timezone is mostly a "UI problem" and should not leak into the backend) but they don't prove your "naive works better" argument.