Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Best Practices for Working with Configuration in Python Applications

tech.preferred.jp

Post reply on HN
Print view View original

21–30 of 69 posts

Re: Best Practices for Working with Configuration in Python Applications

#21

Post by aucontraire »

alanfranz wrote: post #10
aucontraire wrote: post #5

Earlier quoted context omitted.

I agree that python datetime objects are problematic, but for the opposite reason. It is tzinfo that is the sneaky disaster, the plain datetimes are fine. Transparent timezone awareness always fail, unless you are 100% certain that a tzaware datetime object will remain uncoverted from the very top to the very bottom of the stack and all the way up again no matter who is reading and what they are doing. For longterm m…

> I agree that python datetime objects are problematic, but for the opposite reason. It is tzinfo that is the sneaky disaster, the plain datetimes are fine. Why naive datetimes should be fine? How are they fine? What do they represent? > For longterm minimization of pain, bugs and effort, you convert datetimes to UTC This COULD work if naive objects had an IMPLIED UTC in their contract - e.g. naive objects are declar…

There is indeed no contract that datetimes without tzinfo are UTC. But there is a contract that they don't have a builtin timezone or DST concept and that you must define handle that separately.

Re: Best Practices for Working with Configuration in Python Applications

#22

Post by aucontraire »

icebraining wrote: post #13
alanfranz wrote: post #10

Earlier quoted context omitted.

> I agree that python datetime objects are problematic, but for the opposite reason. It is tzinfo that is the sneaky disaster, the plain datetimes are fine. Why naive datetimes should be fine? How are they fine? What do they represent? > For longterm minimization of pain, bugs and effort, you convert datetimes to UTC This COULD work if naive objects had an IMPLIED UTC in their contract - e.g. naive objects are declar…

> Why naive datetimes should be fine? How are they fine? What do they represent? They represent the date/time wherever the user is (location-independent). If I want to take a pill every Monday and Thursday at 10am, I don't want to get a notification at 5am just because I moved from the UK to NY.

Or maybe you do want to take the pill at 5am, since your are only there for a few days and it is critical that you maintain an exact 24 hour interval between doses.

As an assembly worker in the timestamp-wrapper-class factory, I am not in a position to try being clever about it. :-)

Re: Best Practices for Working with Configuration in Python Applications

#23

Post by jry1234 »

Kensho has probably the best solution to this problem I've seen so far:

https://github.com/kensho-technologies/grift

Handles typing really well, as well as config defaults and fallbacks, giving you the ability to configure your app a few ways, and fall back on other configs if something isn't specified.

Re: Best Practices for Working with Configuration in Python Applications

#24

Post by alanfranz »

pydry wrote: post #20
alanfranz wrote: post #4

Earlier quoted context omitted.

Your question implies that you don't know about the nuisances of the datetime library :-) (see https://docs.python.org/3/library/datetime.html it's the first paragraph!) Python datetime objects, by design, can be naive or timezone-aware. Timezone-aware datetime objects are OK; they identify a certain instant in time. Naive datetime objects are Python-only abstractions (AFAIK) that don't identify anything in the real…

>What does the library do? Throw an exception? Associate an arbitrary timezone (e.g. UTC)? Associate the local, current timezone? Naive datetime is what datetime.utcnow() returns. UTC is essentially a "default" timezone. I've always thought it made most sense in a library to assume it's UTC.

Your assumption is just a guess. Take a look at the python docs I linked just above here: "A naive object does not contain enough information to unambiguously locate itself relative to other date/time objects. Whether a naive object represents Coordinated Universal Time (UTC), local time, or time in some other timezone is purely up to the program"

Re: Best Practices for Working with Configuration in Python Applications

#25

Post by alanfranz »

icebraining wrote: post #13
alanfranz wrote: post #10

Earlier quoted context omitted.

> I agree that python datetime objects are problematic, but for the opposite reason. It is tzinfo that is the sneaky disaster, the plain datetimes are fine. Why naive datetimes should be fine? How are they fine? What do they represent? > For longterm minimization of pain, bugs and effort, you convert datetimes to UTC This COULD work if naive objects had an IMPLIED UTC in their contract - e.g. naive objects are declar…

> Why naive datetimes should be fine? How are they fine? What do they represent? They represent the date/time wherever the user is (location-independent). If I want to take a pill every Monday and Thursday at 10am, I don't want to get a notification at 5am just because I moved from the UK to NY.

This is LocalDate/LocalTime in java.time/joda.time parlance. But it's a different beast; in fact, you're talking about a repeated action. But if I tell you that on "March 23rd, 2020, 9:50:01am" I did something, what does that mean to you? When did it happen? That's a naive datetime.

It's got its place: but the idea that the API and the usage should be similar to a precise representation of time, as if it the two were interchangeable is... dangerous, and it's the source of a lot of problems with dt in the Python world.

Re: Best Practices for Working with Configuration in Python Applications

#26

Post by frumiousirc »

kingosticks wrote: post #9
hathym wrote: post #8

it's worth looking at python's own configparser[1] before rolling your own. [1] https://docs.python.org/3/library/configparser.html

Isn't that basically the same end result as using json.loads except a different format (that has no actual spec).

JSON does not support comments nor string interpolation. Python ConfigParser language does.

Re: Best Practices for Working with Configuration in Python Applications

#27

Post by mianos »

ungawatkt wrote: post #14
fermigier wrote: post #11

Quick list of Python libraries that help with application configuration: - Python application configuration -> https://github.com/edaniszewski/bison - Configuration with env variables for Python -> https://github.com/hynek/environ_config - Configuration library for python projects -> https://github.com/willkg/everett - Strict separation of config from code -> https://github.com/henriquebastos/python-decouple This is…

Marshmallow. You can use its schema validation for any dict/json, which makes it a nice fit for validating json config files (which mitigates some of the json concerns from the article). Just immediately move the json.reads through a schema validate, build some classes around it for different config files. marshmallow.readthedocs.io/en/

I use a similar library called Schema at https://pypi.org/project/schema/ I love the expressive nature of it. Just this week I was validating both yaml and json configuration data with it.

One thing missing from this article is to always use a proper external key value store for everything but the stage (dev test prod etc) and the connections to the kv store. Config files on disk and even in environment variables suck for anything but the most trivial platforms.

Re: Best Practices for Working with Configuration in Python Applications

#28

Post by pydry »

alanfranz wrote: post #24
pydry wrote: post #20

Earlier quoted context omitted.

>What does the library do? Throw an exception? Associate an arbitrary timezone (e.g. UTC)? Associate the local, current timezone? Naive datetime is what datetime.utcnow() returns. UTC is essentially a "default" timezone. I've always thought it made most sense in a library to assume it's UTC.

Your assumption is just a guess. Take a look at the python docs I linked just above here: "A naive object does not contain enough information to unambiguously locate itself relative to other date/time objects. Whether a naive object represents Coordinated Universal Time (UTC), local time, or time in some other timezone is purely up to the program"

I'd consider any datetimes that have an implicit timezone that isn't or might not be UTC a bug in any system.

This isn't restricted to python. Servers that spit out logs with timestamps, for instance, should be spitting out UTC.

It makes sense to build systems that deal with timezones at the very edges (and sometimes not even then) and use UTC for everything else. It's simpler that way.

Re: Best Practices for Working with Configuration in Python Applications

#29

Post by albi_lander »

ungawatkt wrote: post #14
fermigier wrote: post #11

Quick list of Python libraries that help with application configuration: - Python application configuration -> https://github.com/edaniszewski/bison - Configuration with env variables for Python -> https://github.com/hynek/environ_config - Configuration library for python projects -> https://github.com/willkg/everett - Strict separation of config from code -> https://github.com/henriquebastos/python-decouple This is…

Marshmallow. You can use its schema validation for any dict/json, which makes it a nice fit for validating json config files (which mitigates some of the json concerns from the article). Just immediately move the json.reads through a schema validate, build some classes around it for different config files. marshmallow.readthedocs.io/en/

dataclass_json is also very useful for schema validation. It combines python's native dataclass objects with marshmallow's schema to provide additional functionalities simply through a @dataclass_json decorator on your dataclass.

https://lidatong.github.io/dataclasses-json/

Re: Best Practices for Working with Configuration in Python Applications

#30

Post by lsorber »

fermigier wrote: post #11

Quick list of Python libraries that help with application configuration: - Python application configuration -> https://github.com/edaniszewski/bison - Configuration with env variables for Python -> https://github.com/hynek/environ_config - Configuration library for python projects -> https://github.com/willkg/everett - Strict separation of config from code -> https://github.com/henriquebastos/python-decouple This is…

Dynaconf [1], though I think this article's approach in combination with pydantic is better.

[1] https://dynaconf.readthedocs.io/en/latest/

Post reply on HN