Markdown is a complete dead end.
I’ve been making a lightweight markup language of my own, and I thought long and hard about this kind of thing, with the goal of making something extremely consistent and easily parseable by human and machine alike. (All popular LMLs are surprisingly hard to parse correctly, so that text editors never have fully correct syntax highlighting unless they use something like LSP-backed highlighting with the real parser.) There’s a common problem with syntax extensions needing semantic understanding before you can actually parse their bodies. I’m using two different syntaxes for the bodies of what I’m calling macros (here shown without arguments to the macros, partly because I’m still not entirely satisfied with any of the syntaxes I’ve tried for them):
@macro-name{interpreted-macro-body}
@macro-name`raw-macro-body`
In the case of an interpreted macro body, it will be parsed fully (supporting both block and inline formatting) and fed to the macro so; in the case of a raw body, it will will be fed to the macro uninterpreted, just like with the `…` monospace code syntax. (If you wanted to pass a monospace code element as the body, that’d be @macro-name{`…`}. In the general context, the monospaced code syntax `…` is basically just shorthand for @code`…`, like **bold** can be shorthand for @bold{bold}.) This would lead to the shortest possible syntax for mathematics being @m`…`, which I think is acceptable, and much more syntactically robust. If you needed backticks inside the body, you’d currently have to use @m{…} syntax, backslash-escaping any special syntax, because I haven’t come up with any satisfactory other syntax (allowing delimiter repetition, like @m```…```, doesn’t solve all cases as you can’t use the delimiter at the start or end of the value, a problem that most LMLs that go this way seem to ignore, e.g. I think there are some things that you genuinely can’t express in reStructuredText because of this, and others have awful syntactic hacks like backslash space being special; I’m contemplating @m#`…`# and @m#{…}# with arbitrary but matching number of hashes, like Rust’s raw strings, but it’s still not as neat, so I could end up just leaving it at “use an interpreted body and escape everything”). All up, I think this raw/interpreted body distinction should work pretty well, and is sound.