Not exactly "experimental", considering the Unix heritage, but -- line editors.
"I've seen [visual] editors like that, but I don't feel a need for them. I don't want to see the state of the file when I'm editing." -- Ken Thompson, on the superiority of ed to visual editors. Summarized by Peter Salus in A Quarter Century of UNIX (Addison-Wesley, 1994).
Definitely a blast from the past, but I do think line editors may force one to write simpler programs -- or to think in smaller chunks, as opposed to (doom)scrolling or moving about incrementally on a large screen. Who knows, maybe the line editing approach is making a comeback of sorts, with people figuring out how to write code using tablets, smartphones, RasPi-based low-powered devices, etc. Maybe we will start re-thinking the (possible) mental benefits of smaller screens / viewing less lines of text or code at a time.
Rob Pike's sam editor has an interesting command language. You're not limited to thinking in "lines" as in ed or sed; rather, the whole file is a giant string that you manipulate using regular expressions, external pipes, etc: http://doc.cat-v.org/bell_labs/sam_lang_tutorial/sam_tut.pdf
In the light of the OP's question, I suppose Pike's "structural regular expressions" are the keyword here (explained in the above linked paper). It is a paradigm worth pointing out; also used later in screen-oriented editors like vis.
sam's predecessor, qed, is also interesting, extremely powerful, but it seems to have a much steeper learning curve. I have used sam quite a bit, but not qed. https://github.com/phonologus/QED/raw/master/doc/qed-tutoria...