Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Common Expression Language

github.com

Post reply on HN
Print view View original

71–80 of 111 posts

Re: Common Expression Language

#71

Post by dorfsmay »

jhhh wrote: post #52
derefr wrote: post #42

Earlier quoted context omitted.

What about the option of just writing a regular, one-off program in a regular programming language , the output of which is your baked YAML config; and then having a pipeline that involves running that config-generator program, piping its output to your orchestrator of choice? Nearly every programming language has a YAML serialization library†. And before that serialization happens, your config can be expressed using…

After trying to write complex loop statements and conditionals in yaml for ansible I had this thought as well. It's nice when declarative configurations work, but once they don't and you have to try to write a real program in yaml you'll want to pull your hair out.

This was my exact experience as well. I was an early user if sensible and loved that it was yaml. Then had to deal with jinja inside yaml. And finally a syntax for loops appeared!

I have run into the same type of issues with salt.

Re: Common Expression Language

#72

Post by hardwaresofton »

tonyhb wrote: post #67
hardwaresofton wrote: post #9

This looks really similar to Open Policy Agent[0], wonder how they compare. [0]: https://www.openpolicyagent.org/

I use both in different projects. OPA (and its language Rego) is a good matching and policy engine with declarative blocks, modules, and expressive functions for HTTP headers, JWTs, etc. It's great for security. For building up abstractions and testing arbitrary JSON with complex, pre-defined policies. Testing is built in, and that's great. You can create "functions" and your own DSL for matching/evaluation. If you w…

Thanks for breaking this down -- that makes a ton of sense. I've sene OPA's use along with k8s but haven't seen much use of it outside k8s yet. It seems like almost a special case.

My biggest problem with a lot of these generic computation (you could view OPA as generic computation but with a focus on auth) is that they bring their own DSLs -- I'd love to see something like CEL that's based on regular programming languages, and the only way I can think of doing that right now is through WASM.

Re: Common Expression Language

#73

Post by derefr »

lmm wrote: post #65
derefr wrote: post #62

Earlier quoted context omitted.

> At that point why use YAML at all? Ideological answer: For the same reason HTTP/2.0’s binary protocol didn’t instantly obviate/deprecate HTTP/1.0’s text protocol. Text has advantages: text is debuggable, and prototypable. If the interface between two programs is a text based declarative language, you can audit that text, diff that text, edit that text to see how changes affect the result, mock one side or the other…

> Ideological answer: For the same reason HTTP/2.0’s binary protocol didn’t instantly obviate/deprecate HTTP/1.0’s text protocol. Text has advantages: text is debuggable, and prototypable. If the interface between two programs is a text based declarative language, you can audit that text, diff that text, edit that text to see how changes affect the result, mock one side or the other by producing or consuming that tex…

But the third-party tool frequently isn’t intended to (only) consume machine-generated config. It’s usually built to consume a format that could equally be machine-generated or hand-authored. Usually with an emphasis on hand-authoring, where machine-generation is an automation over hand-authoring that will only need to happen as one scales; and so high-complexity machine-generation will only be relevant to the most enterprise-y of integrators.

Other examples of formats like this, that are hand-authored in the small but generated in the large: RSS, SQL, CSV.

Again, Kubernetes is a prime example of this. K8s config YAML is designed with the intention of being hand-authored and hand-edited. It’s only when devs or their tools need to auto-generate entire k8s cluster definitions, that you begin needing to machine-generate this YAML. This generated YAML is expected to still be audited by eye and patched by hand after insertion, though, so it still needs to be in a format amenable to those cases, rather than in a format optimal for machine consumption.

> if we're generating this config with code then we don't want to diff or edit the generated config

Look more into GitOps. The idea behind it is that whatever tooling you’re using to generate config is run and the resulting config is committed to a “deployment” repo as a PR; ops staff (who don’t necessarily trust the tooling that generated the config) can then audit the PR, and the low-level changes it describes, before accepting it as the new converged system state. It puts a human veto in the pipeline between machine-generated config and continuous deployment; and allows for debugging when upstream tweaks aren’t having the low-level side-effects on system state one would expect.

Re: Common Expression Language

#74

Post by stevekemp »

I've never used this particular library, but I did put together my own simple evaluation engine and have found it very useful for a range of purposes.

Initially it was designed to process incoming slack messages, and sometimes trigger a notification to an on-call engineer, but over time I've found uses for it processing email, scripting simple actions on my desktop, and more.

https://github.com/skx/evalfilter/

These kind of things are pretty simple to write, but sometimes I almost think it is a shame there isn't something more standard. (Lua was kinda winning for that embedded-logic role for a long time, but nowadays we still have the mixture of YAML, HCL, and other niche-specific language/filtering and I imagine the time has passed to pick one standard.)

Re: Common Expression Language

#75

Post by stevekemp »

lmm wrote: post #56
derefr wrote: post #42

Earlier quoted context omitted.

What about the option of just writing a regular, one-off program in a regular programming language , the output of which is your baked YAML config; and then having a pipeline that involves running that config-generator program, piping its output to your orchestrator of choice? Nearly every programming language has a YAML serialization library†. And before that serialization happens, your config can be expressed using…

At that point why use YAML at all? If it's generated by a program and fed to a program, you're better off using protobuf or something like that. In fact, since you're probably using the same language on both ends, why not just write a regular value in your language? This probably sounds like a strawman, but it's not. It's how a lot of e.g. Python projects are configured - the "config" file is just a normal bit of cod…

Things like AWS Cloudformation require YAML input, so there's no real choice on what you emit.

But writing the YAML is fiddly and annoying, so that's a good example of something where it is better to generate it via troposphere (a python module) or some similar system.

To be less specific I guess the answer is that sometimes you don't control both ends - the part that emits and the part that consumes, and having faught ansible, and similar tools, if I can avoid it I'd never want to write YAML by hand for non-trivial purposes if I could script it instead.

Re: Common Expression Language

#76

Post by mst »

KMag wrote: post #61
frou_dh wrote: post #5

I remember seeing this used inside config files for the Caddy webserver. Google seem to like these executable config languages because they've got another open source one ("Starlark") a few notches up in expressivity.

"Executable" configuration languages (most "non-executable" configuration language parsers are push down automatons that execute the configuration) are handy, but without strong coding standards and good discipline, the line between business logic and configuration tends to blur over time. Cartesian product, map, and reduce operations over finite sets and lists are really handy in configuration. ("For each server in…

> When the only tool some of your people are allowed to use is a hammer, hammer marks start mysteriously showing up everywhere.

This ... yes. Being the ops guy backing up second line support at an ISP for a while brought me many examples of this and inspired many in-house tools for them to use instead.

Re: Common Expression Language

#77

Post by SgtPinback »

codebje wrote: post #51
progval wrote: post #27

Earlier quoted context omitted.

> Although you can get it to solve 3-SAT, though only for some predefined number of variables (which it assures can be at least 32). Combinatorics stuff like printing all possible sudokus also seems like it should be feasible. How is this compatible with their claim that "CEL evaluates in linear time"?

The only place I can find that makes the 3-SAT claim is this comment thread. Nevertheless, there's two answers I can think of here. The first is simple (but IMO probably not right): 3-SAT is (probably) exponential, but the language definition document qualifies the linear claim by saying that macro expansion can also be exponential. The second is probably more accurate, though. General 3-SAT is (probably) exponential…

Hi, I'm Jim (@JimLarson), another CEL maintainer. Yes, the full language spec is more clear about the performance limits, and macros can easily result in exponential time or space complexity. The original claim could be for either the macro version or the equivalent expanded version, e.g. "[true, false].exists(v1, [true, false].exists(v2, ... [true, false].exists(vN, (v1 || !v2 || v3) && (v2 || v3 || !v4) && ...)...))", etc. But it doesn't require much power in the language to express this kind of "solver" - there's still no general recursion.

Re: Common Expression Language

#78

Post by otabdeveloper4 »

didip wrote: post #10
throwaway894345 wrote: post #8

Earlier quoted context omitted.

I'm an ardent supporter of executable config languages, especially for the infrastructure-as-code space (the only thing special about this space is that configs tend to be very large, so you're more likely to run into reuse issues), which markets itself as "it's just YAML!" but inevitably all of that copy/pasted YAML becomes unwieldy and you want reusability. At that point, you have a few distinct options: 1. Build a…

I am with you, I wish the world would just adopt Lua as their config files.

Then you'll need config files for your config files, and we're back to square one.

The scope and capabilities of the config language needs to be limited, otherwise we lose the ergonomic benefits of configuration in the first place.

Re: Common Expression Language

#80

Post by pmontra »

lmm wrote: post #56
derefr wrote: post #42

Earlier quoted context omitted.

What about the option of just writing a regular, one-off program in a regular programming language , the output of which is your baked YAML config; and then having a pipeline that involves running that config-generator program, piping its output to your orchestrator of choice? Nearly every programming language has a YAML serialization library†. And before that serialization happens, your config can be expressed using…

At that point why use YAML at all? If it's generated by a program and fed to a program, you're better off using protobuf or something like that. In fact, since you're probably using the same language on both ends, why not just write a regular value in your language? This probably sounds like a strawman, but it's not. It's how a lot of e.g. Python projects are configured - the "config" file is just a normal bit of cod…

This is even more true for Ruby. The language is famous for the ease of creating DSLs because of block passing and optional parentheses. Examples: puppet, chef, vagrant, Rails' configuration files. I still remember the joy of not configuring a project with XML coming from Java Structs in 2006.
Post reply on HN