Earlier quoted context omitted.
Things like AWS Cloudformation require YAML input, so there's no real choice on what you emit. But writing the YAML is fiddly and annoying, so that's a good example of something where it is better to generate it via troposphere (a python module) or some similar system. To be less specific I guess the answer is that sometimes you don't control both ends - the part that emits and the part that consumes, and having faug…
Just write JSON and pretend it's YAML. YAML is a superset of JSON so there's no need to generate "nice" YAML if there isn't a human reading or writing it.
Common Expression Language
91–100 of 111 posts
Re: Common Expression Language
#92Earlier quoted context omitted.
But the third-party tool frequently isn’t intended to (only) consume machine-generated config. It’s usually built to consume a format that could equally be machine-generated or hand-authored. Usually with an emphasis on hand-authoring, where machine-generation is an automation over hand-authoring that will only need to happen as one scales; and so high-complexity machine-generation will only be relevant to the most e…
In most programming languages you can hand author a value just fine - that part isn't an advantage to something like YAML or json. Given the use of variables and a few other similar simple techniques, I dare say many programming languages are more amenable to hand-authoring static config objects than most static config languages. I think the real issue is reproducibility; and that boils down to purity. Fully fledged…
But keep in mind that we’re not inherently talking about programming languages here — nor are we necessarily talking about people capable of programming as our configurators. We’re talking about third-party components that need to be configured by ops people, who may or may not be DevOps people. Usually they’re not — most ops people are just pure ops, and don’t know any programming languages. As well, most amateur integrators (e.g. a person setting up their own blog) aren’t programmers either.
The goal of these systems, when choosing a configuration solution, is twofold: to give pure-ops and amateur integrators a config language they can author directly, in a text editor, without learning programming; while also making that language formal/structured enough that it’s easy to machine-generate from your programming runtime of choice, if you do have those skills, and a rigorous mindset.
Sure, programming languages don’t necessarily require you to use the full-fledged expression syntax they enable, and so can “reduce” to a configuration-language-like subset of themselves.
But remember, again — ops people and amateur integrators. What do such people tend to do, to create their config? Read the reference config schema? No. They tend to look up tutorials with samples, or StackOverflow “solutions”, from arbitrary places on the Internet.
And what do the creators of those samples have in abundance? Cleverness and a desire for clarity of meaning. Traits that cause them to use the expressive features of whatever the configuration language is, in order to make their answers more “pithy”.
Which means that, to wield these “pithy” samples/solutions, the ops people and amateur integrators now have to understand how to “patch” one arbitrary piece of complex code into another increasingly-arbitrary piece of complex code.
The thing a static data-serialization format gets you, is that the rules for merging any two expression-nodes in it are very simple to learn, because there just aren’t that many types of expressions. There’s no way to be “pithy” with the configuration that requires people to learn entirely-new-to-them syntax.
By choosing to configure your system in YAML, you’re guaranteeing that the samples these ops people and amateur integrators find and attempt to glue together, will also just be pure YAML. And since their existing config file, and each new sample, are pure YAML, they’ll likely succeed at doing this gluing-together.
Meanwhile, DevOps people and enterprise integrators can create their own programs to generate the YAML — but since there’s no first-party framework for doing this, there won’t be much value in sharing these programs around, and so the samples the pure-ops people and amateur integrators find will never be given “in terms of” writing code for such a framework, but rather only in terms of the config YAML itself.
> I think the real issue is reproducibility; and that boils down to purity. [...] If you can rigorously avoid that, there's not too much advantage to a static config language.
Individual users might be able to rigorously avoid that (though expecting a rigorous approach to formal expression from non-programmers is a bit much.) But often it's the system itself that needs purity and reproducibility.
Remember, config formats are usually something executed at every startup — in other words, they're durable state that happens to be human-modifiable. (Think: the Windows Registry.) As the designer of a system, you don't want the same state you serialized today to deserialize to something else tomorrow; and you especially don't want the meaning of your state to depend contextually on the environment. You want to "pin down" your state.
A good example: programming-language package-ecosystem "lock files." In most languages, dependency-constraint specification is done in a programming language, such that the generation of those constraint expressions has access Turing-complete features. But once you lock those constraints down to a baked set of choices, the lockfile itself — the predetermined set of choices, that should be environment-independent — is not expressed in a Turing complete language (in any runtime I know of, at least) but rather is always expressed in its own little static declarative language; or at most in a limited "data-expressions only" subset of the parent language (e.g. Erlang's `file:consult/1` format.)
In this case, dep-constraints are the inputs to a config-generator program; while the lockfile is the config format itself. The config format is a necessary intermediate here; it'd be impossible for the runtime to make the same static guarantees about package management if it wasn't! (In fact, see e.g. Python's setup.py, where exactly that problem stymies any package-manager the Python ecosystem introduces from pre-determining dependency graphs before actually downloading and attempting installation of the dependencies.)
Re: Common Expression Language
#93Re: Common Expression Language
#94If you’re constantly fighting yaml, consider jsonnet. It’s another project by google and similarly not Turing complete. Works wonderful at generating templates.
Re: Common Expression Language
#95> Expr = ConditionalOr ["?" ConditionalOr ":" Expr] ; I believe this was a mistake. "?" based conditionals aren't really a good idea IMHO.
Re: Common Expression Language
#96Re: Common Expression Language
#97Fast - CEL runs without the need for sandboxing, making it much faster than sandboxed solutions like WebAssembly, Lua, and embedded JavaScript.
Scalable - Features like variables and functions would make CEL more expressive, but also less scalable as it's easy to write a few lines of code with functions that consume exponential amounts of memory and compute. CEL is simply the expression and nothing more.
Portable - CEL is implemented in Go[0], C++[1], and Python[2] with Java open sourcing in development. There is a public codelab[3] available for Go if anyone is interested. There is also a conformance suite in CEL-Spec to ensure consistent behavior between runtimes and environments. Our objective is to make it possible to bring CEL to K8s, J2EE apps, and C++ proxies. Evaluate at line-rate everywhere. Personally, I hope someone tries to make CEL work on IoT devices some day too.
Where? - CEL is usually embedded into larger projects rather than being the one stop shop for solving a particular kind of problem. For example, CEL Policy Templates[4] has an opinionated way of using CEL to validate/evaluate YAML configs. Most of the time CEL is part of a service API.
In addition to being used in Firebase's Cloud Firestore / Cloud Storage security rules, it is also used in several other Google Cloud services: - Cloud Armor[5] - IAM Conditions[6] - Cloud Healthcare Consents[7] - Cloud Build Notifiers[8] - Security Token Service[9] - Access Levels[10], and more.
CEL is also used in some prominent open source projects like Envoy RBAC[11], Caddyserver[12], Krakend.io[13], and Cloud Custodian[14].
[0]: https://github.com/google/cel-go [1]: https://github.com/google/cel-cpp [2]: https://github.com/cloud-custodian/cel-python [3]: https://codelabs.developers.google.com/codelabs/cel-go [4]: https://github.com/google/cel-policy-templates-go [5]: https://cloud.google.com/armor/docs/rules-language-reference [6]: https://cloud.google.com/iam/docs/conditions-overview [7]: https://cloud.google.com/healthcare/docs/concepts/consent-mo... [8]: https://cloud.google.com/cloud-build/docs/filter-build-notif... [9]: https://cloud.google.com/iam/docs/workload-identity-federati... [10]: https://cloud.google.com/access-context-manager/docs/custom-... [11]: https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overv... [12]: https://caddyserver.com/docs/caddyfile/matchers#expression [13]: https://www.krakend.io/docs/endpoints/common-expression-lang... [14]: https://github.com/cloud-custodian/cel-python
Re: Common Expression Language
#98Googler here: for those of you who have ever used Firebase this is the language that powers Cloud Firestore / Cloud Storage security rules. Specifically in our rules everything after the "if" is Common Expression Language. See: https://firebase.google.com/docs/firestore/security/rules-co... The efficiency and safety of CEL enables us to put security rules in the critical path of every database request.
“this is the language that powers Cloud Firestore / Cloud Storage security rules” This really needs to be stated on its front page. Right now it’s all “Hows” and no “Why”. First question anyone looking at it asks: What problem does it solve?/What need does it fill? A real-world use case provides an easy relatable answer. Incidentally, with existing links to protobuf and no halting problem to worry about, it sounds li…
Re: Common Expression Language
#99I can't escape the feeling that emacs got this right. Nobody wants their config to be lisp, but it fits the bill for what you needed. Especially combined with the custom sections. So nice.
Your config language being turing complete doesn't work if you need to accept/validate config files from untrusted parties
Re: Common Expression Language
#100> Expr = ConditionalOr ["?" ConditionalOr ":" Expr] ; I believe this was a mistake. "?" based conditionals aren't really a good idea IMHO.
Go on... :-). Why don't you like them?
Instead of ` ? : ` I prefer `if { } else { }` the additional brackets noticeable improve readability and you can extend it to support `if { } else if { } else { ` instead of ` ? : ? : `. (Oh and that last example might be wrong needing brackets depending on operator precedence...)
Through if you don't nest it it doesn't matter (oh and because it's a expression evaluation `else` is not optional but required as you need a value the expression resolves to).