Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Best Practices for Working with Configuration in Python Applications

tech.preferred.jp

Post reply on HN
Print view View original

51–60 of 69 posts

Re: Best Practices for Working with Configuration in Python Applications

#51

Post by kingosticks »

nomel wrote: post #46
kingosticks wrote: post #32

Earlier quoted context omitted.

True, you do get string interpolation but the comment support in ConfigParser isn't very good. Although actually they may have fixed some of that in Python 3 but I'm still using workarounds. To be clear, I am not suggesting using JSON for config, I think that would be my last choice. My point is that ConfigParser isn't really an alternative to rolling your own if you want decent validation etc (those spec files are h…

What's wrong with the comment support? You can't have comments at the end of a line, but that's sort of the nature of supporting arbitrary strings as values. I don't want my users to have to quote or escape special characters if they happen to want to use them. They're not programmers. # The note to display note = Our #1 customer! rather than name = Our \#1 customer # The note to display. or name = "Our #1 customer"…

> What's wrong with the comment support?

Comments are simply ignored. You can't read them. You might want to read a commented config file in, make a change to a setting and then write that out. You can't do that. But you can write comments using the 'allow_no_value=True' hack, as long as long as you put it in a section.

> You can't have comments at the end of a line

You can. You need to use ';' for inline comments and you must proceed it with whitespace. Are your users ready for that?

Re: Best Practices for Working with Configuration in Python Applications

#52

Post by _pastel »

As an ML engineer working in Python, I keep running into a problem:

If parameters are defined close to usage, and strongly typed, then it's hard to cleanly search for good configurations of the parameters. Especially for fancier search strategies, you want all parameter lookups to go through a single file.

On the other hand, there's a lot of code churn until an ML pipeline is finished. And errors from typos and type violations will often only show up after hours of training. So it's also painful to try to keep a separate, loosely-typed parameter file in sync.

So far, my compromise is to:

(1) on a first pass, define all parameters as global variables at the top of the files they are used in

(2) once mostly code-complete, pull them into a separate file that tracks initial values, current values, and search ranges. Make all usages go through a lookup where the key is an enum, but the value is untyped:

  def param(name: ParamName) -> Any:
    return params[name].current_value
Which is not ideal. Does anyone else keep running into this problem and have a better solution?

Re: Best Practices for Working with Configuration in Python Applications

#53

Post by silviogutierrez »

_pastel wrote: post #52

As an ML engineer working in Python, I keep running into a problem: If parameters are defined close to usage, and strongly typed, then it's hard to cleanly search for good configurations of the parameters. Especially for fancier search strategies, you want all parameter lookups to go through a single file. On the other hand, there's a lot of code churn until an ML pipeline is finished. And errors from typos and type…

Alas, MyPy doesn't have the concept of "keyof" and mapped types like TypeScript.

So in place of that I would:

1. Define a variable called Params = Any;

2. Liberally use Param["foo"] and Param["bar"] anywhere.

3. Once you stabilize, reimplement Params as a TypedDict. You'll get failures if accessing any invalid key.

You can also use a NamedTuple if you prefer.

If you insist passing a param name, then you'll have to create a big list of Literals for param with every key in your dict. So Param = Literal["foo", "bar"] etc.

Re: Best Practices for Working with Configuration in Python Applications

#54

Post by _pastel »

silviogutierrez wrote: post #53
_pastel wrote: post #52

As an ML engineer working in Python, I keep running into a problem: If parameters are defined close to usage, and strongly typed, then it's hard to cleanly search for good configurations of the parameters. Especially for fancier search strategies, you want all parameter lookups to go through a single file. On the other hand, there's a lot of code churn until an ML pipeline is finished. And errors from typos and type…

Alas, MyPy doesn't have the concept of "keyof" and mapped types like TypeScript. So in place of that I would: 1. Define a variable called Params = Any; 2. Liberally use Param["foo"] and Param["bar"] anywhere. 3. Once you stabilize, reimplement Params as a TypedDict. You'll get failures if accessing any invalid key. You can also use a NamedTuple if you prefer. If you insist passing a param name , then you'll have to c…

Thank you! It's surprising that mypy can typecheck based on string keys like that. Cool!

Re: Best Practices for Working with Configuration in Python Applications

#55

Post by rb808 »

fermigier wrote: post #11

Quick list of Python libraries that help with application configuration: - Python application configuration -> https://github.com/edaniszewski/bison - Configuration with env variables for Python -> https://github.com/hynek/environ_config - Configuration library for python projects -> https://github.com/willkg/everett - Strict separation of config from code -> https://github.com/henriquebastos/python-decouple This is…

This and the list of other libraries in responses is a great reason to avoid Python. Its nearly as bad as Javascript, how are you supposed to live with this.

Re: Best Practices for Working with Configuration in Python Applications

#56

Post by nomel »

kingosticks wrote: post #51
nomel wrote: post #46

Earlier quoted context omitted.

What's wrong with the comment support? You can't have comments at the end of a line, but that's sort of the nature of supporting arbitrary strings as values. I don't want my users to have to quote or escape special characters if they happen to want to use them. They're not programmers. # The note to display note = Our #1 customer! rather than name = Our \#1 customer # The note to display. or name = "Our #1 customer"…

> What's wrong with the comment support? Comments are simply ignored. You can't read them. You might want to read a commented config file in, make a change to a setting and then write that out. You can't do that. But you can write comments using the 'allow_no_value=True' hack, as long as long as you put it in a section. > You can't have comments at the end of a line You can. You need to use ';' for inline comments an…

> make a change to a setting and then write that out

Very good point.

> You can. You need to use ';' for inline comments

I have some bugs to fix.

Re: Best Practices for Working with Configuration in Python Applications

#57

Post by alanfranz »

thomk wrote: post #42
alanfranz wrote: post #2

While most points are valid, I feel some pieces are missing in this article. What should I finally do? How to put everything together without creating an hard-to-maintain mess of casting/parsing/configuration? Should I manually cast strings to integers (or other types) for all and each value I parse? Where do I keep default values? It's cumbersome to have them embedded in code for get(). I usually want a) a default c…

Who is the targeted editor of a config file? If it's another programmer, just use config.py and make life easy on yourself.

No. Using a programming language for configuration is terrible. If code can be used, it will be eventually used and it will be impossible to understand what's happening until runtime. Configuration should be clear and not subject to modifications to the software. If somebody wants to use an external preprocessing tool, if using a standard format (json) he/she can.

Re: Best Practices for Working with Configuration in Python Applications

#58

Post by alanfranz »

brewmarche wrote: post #38
aucontraire wrote: post #5

Earlier quoted context omitted.

I agree that python datetime objects are problematic, but for the opposite reason. It is tzinfo that is the sneaky disaster, the plain datetimes are fine. Transparent timezone awareness always fail, unless you are 100% certain that a tzaware datetime object will remain uncoverted from the very top to the very bottom of the stack and all the way up again no matter who is reading and what they are doing. For longterm m…

Actually UTC + timezone is exactly the wrong thing for "wall clock times" (things like meetings or departures where the time at the location is relevant). The conversion to UTC will lose the original local time so you cannot retrieve it once time zone data changes, unless you perform reconversions every time you detect such a change in tzdata. And countries changing time zones happens more often than we think (and al…

> The conversion to UTC will lose the original local time so you cannot retrieve it once time zone data changes, unless you perform reconversions every time you detect such a change in tzdata

I don't agree/disagree with your point, and neither I do agree/disagree with GP on the topic, but why couldn't I retrieve the original time? If, for an event, I save UTC + event TimeZone, I can always get back to the original time (actually, it doesn't even matter whether the timestamp is UTC; it's enough for it to have an explicit offset, i.e. to be the representation for an instant). Why should I change the timezone on a saved record? What usually changes is the user's timezone, not the records'.

> Thus it is important to distinguish between instants and wall clock time

Yes.

> For more information Jon Skeet has written about this multiple times.

I have read many things on datetime; would you care to share a couple of relevant links?

Re: Best Practices for Working with Configuration in Python Applications

#59

Post by thomk »

alanfranz wrote: post #57
thomk wrote: post #42

Earlier quoted context omitted.

Who is the targeted editor of a config file? If it's another programmer, just use config.py and make life easy on yourself.

No. Using a programming language for configuration is terrible. If code can be used, it will be eventually used and it will be impossible to understand what's happening until runtime. Configuration should be clear and not subject to modifications to the software. If somebody wants to use an external preprocessing tool, if using a standard format (json) he/she can.

Configuration, by definition, modifies the software.

How much it modifies the software is determined by the software author. If there aren't limits imposed in your example (json) you're still going to get unexpected results. You'll have to limit what's accepted.

Also, what if I allow a config.py file then translate that to .json then back to python? That is no better than just allowing the consumption of a python source file and simply ignoring any disallowed code (again, limiting what is accepted).

I know it's vogue to say configuration as code is bad; that simply has not been my experience.

Re: Best Practices for Working with Configuration in Python Applications

#60

Post by alanfranz »

thomk wrote: post #59
alanfranz wrote: post #57

Earlier quoted context omitted.

No. Using a programming language for configuration is terrible. If code can be used, it will be eventually used and it will be impossible to understand what's happening until runtime. Configuration should be clear and not subject to modifications to the software. If somebody wants to use an external preprocessing tool, if using a standard format (json) he/she can.

Configuration, by definition, modifies the software. How much it modifies the software is determined by the software author. If there aren't limits imposed in your example (json) you're still going to get unexpected results. You'll have to limit what's accepted. Also, what if I allow a config.py file then translate that to .json then back to python? That is no better than just allowing the consumption of a python sou…

I don't know whether it's vogue or not, I think it's just a bad idea, because you can't restrict HOW MUCH is is modifying the software, and then people needs to know the nuisances of your programming language. What if I need some configuration to be user (not admin) configurable? What if I don't want to learn programming language X just to use a software?

> If there aren't limits imposed in your example (json) you're still going to get unexpected results.

It's very unlikely for a JSON string to be parsed as a class, or to execute any code, unless you do eval() or do unsafe deserialization. A Python config could do anything.

> You'll have to limit what's accepted.

This is much, MUCH easier said than done. How would you do that in Python? Google App Engine had a sort of "restricted python" idea, and it was hard to implement AFAIK. Same thing for Zope/Plone (there was some templating with restrictions, don't remember the precise system). Then, you'd need do document such "restricted python".

> Also, what if I allow a config.py file then translate that to .json then back to python?

I suppose the first config.py is under the control of the user, and the second is under control of a software author. This is ok, because the "second python version" can perform validation and object construction.

Python configs can be OK (but can get messy) if the software you're building is mostly internal, and few people modify it and they all know what they're doing. As soon as you've got a large enough team and or userbase, IMHO executable configurations are painful.

Post reply on HN