Red Programming Language: Plans for 2019

> Red/Rebol might actually be too ad hoc

No offence, but very few people who raise such concerns actually take time to learn the language (or even launch a REPL at least once) and understand its design, so I appreciate you digging deeper. "Things are thrown against the wall" only in terms of constant search for a sound business model, and this, I believe, is the struggle that any startup (esp. programming languages) faces.

> Is there means to access the lexer directly as a means to explicate how, exactly, Red is interpreting?

I think you can start by, well, reading lexer code, which is written in Parse dialect [1], but specification I showed you might be more approachable. But really, just grab the latest build and start playing, I'll give some very basic examples below.

Now, to the main point: from what I know, Rebol (and Red) are based on research in denotational semantics that Carl Sassenrath did. I'll try to briefly explain the main points.

As you already know, everything starts with a UTF-8 encoded string. Each valid token in this string is converted to an internal data representation - a boxed structure, called a value slot or sometimes a cell.

Value slot is composed of a header and a payload. Header contains various flags and datatype ID, payload specifies exact content of the value. If content doesn't fit in one value slot, then payload contains a pointer to an external buffer (an array of value slots, bytes, or other units + offset and start/end addresses IIRC) with extra data.

So, lexer converts string representation to a tree of value slots (this phase is called "loading"), which is essentially a concrete syntax tree (CST) - this is the crux of homoiconicity.

  >> "6 * 7"
  == "6 * 7"
  >> type? "6 * 7"
  == string!
  >> load "6 * 7"
  == [6 * 7]
  >> type? load "6 * 7"
  == block!
  >> first load "6 * 7"
  == 6
  >> type? first load "6 * 7"
  == integer!
 

Everything is a (first-class) value, and every value has a datatype (we have roughly 50 of them right now). And there's no code - only this data structure, which is just a block, which you can freely manipulate at will (so as any other value).

  >> reverse [6 * 7]
  == [7 * 6]
  >> append reverse [6 * 7] [+ 1]
  == [7 * 6 + 1]
  >> skip append reverse [6 * 7] [+ 1] 2
  == [6 + 1]

What interpreter does is just a "walk" over this tree of values, dictated by a set of simple evaluation rules (expressions are evaluated left to right, operators take precedence over functions and have a more tight left side, literals evaluate to themselves, functions are applied to a fixed set of arguments, symbolic values of type "set-word!" [more on this later] are bound to the result of expression that follows them, etc) but there are a couple of catches.

The first catch is that some values are symbolic - that is, they indirectly refer to some other values via a context (namespace). You can modify this reference (called binding) freely at runtime, and thus change the meaning of symbolic values and of an entire block that contains them.

So, the "meaning" of a given block is always relative to some context(s) - this is what relative expression means (RE in REBOL). And context itself is just an environment of key/value pairs (key is a "symbol", value is its "meaning") represented as an object (O in REBOL).

  >> block: [6 * 7]             ; "block:" is a value of type "set-word!"
  == [6 * 7]
  >> type? second block
  == word!                      ; words are one of the symbolic values I've mentioned
  >> do block
  == 42
  >> bind block object [*: :+] ; now "multiplication" means "addition"
  == [6 * 7]
  >> do block
  == 13

The second catch is that you are not restricted by default interpreter (represented by "do" function) and can use any other one or even implement your own, thus making an embedded DSL - a dialect in Red/Rebol parlance.

* Red/System takes a block of C-level code and does the prescribed job.

* View takes a block that specifies GUI layout and shows a fancy window.

* Draw takes a block of drawing commands and renders an image.

* Parse takes an input series and a block of PEG grammar, and parses the input.

* Math takes a block and interprets it with common operator precedence rules.

Sky is the limit, and its dialects all the way down.

To reiterate: the basic building block (no pun intended) is a block of values, which can represent either code (relative expression which, upon evaluation, will yield a value) or data (just a bunch of values arranged in a specific format - such micro-formats are considered to be dialects too †). Block can also contain symbolic values (called words) which can change their binding during evaluation, and thus alter the semantics of expression.

There's a lot hiding behind the facade, as you can see. And what is there is hardly an ad-hoc hodge-podge slapped together.

[1]: https://github.com/red/red/blob/master/environment/lexer.red

(†): one example of such micro-format dialect is function specification, e.g.

  spec: [
      "Add two numeric values together"
      x [number!]
      y [number!]
  ]

is a specification (or an "interace") for a function that performs addition. Here's a block that expresses addition of two specific numbers:

  [1 + 2]

If we wish to abstract over it, we can substitute 1 and 2 for words:

  expression: [x + y]

And then we can alter bindings of these words to actual arguments we wish to add together:

   bind expression object [x: 1 y: 2]

We then can evaluate such expression and yield a resulting value:

   == do expression
   >> 3

The trick is that functions are just abstraction over evaluation of expression in some environment, that is, a syntax sugar for

  do bind [...] object [...]

with some additional optimizations and type-checking. So, addition instead can be expressed as:

  >> add: func spec expression
  == func [
    "Add two numeric values together" 
    x [number!] y [number!]
  ][x + y]
  >> type? :add ; ":add" is a value of type "get-word!" which, on evaluation, yields function's value referred by word "add" as-is, without triggering its application.
  == function!
  >> add 1 2
  == 3

@hjek, as @dockimbel noted, and we have never concealed, Red was bootstrapped using Rebol. Doc's stats break things down pretty well, but there's another point to be made. Red is ~95% Rebol compatible. This is a very important detail, and proved that Red could be used to self host in the future. When the need to self-host becomes important, it can be done relatively easily, and with confidence.

Is that a fair and accurate assessment?

> And use official resources (Red and Rebol websites) as references, would you? Wiki page is an extremely weak testimony to whatever you have to say.

That's my mistake. The quote about Clojure I provided were from `redprogramming.com` which is linked to from the official website and from your Github, so it looked official to me.

The comparison with Python is from a blog of one of the Rebol developers. Not sure whether that qualifies as official.

If the Wikipedia article about Red is wrong, it might be a good idea to correct it..?

> Even if there is a cost in using Rebol2 as bootsrapping language, it will be gone with 1.0 release. [...] (since when bootstrapping language is a dependency?)

That's cool, and I wasn't aware of that. I just assumed Red was to Rebol what Hy is to Python and Clojure to Java, but that's great that that is a non-issue!

> The use of Rebol is only for boostrapping the language. The whole Rebol code part will be dropped after 1.0, and the toolchain rewritten in pure Red.

Cool!

> I'd like to be able to hand a string (or file) of what I think is valid (or invalid) Red code and have the lexer return to me that input NOT executed but parsed into lexemes, grouping tokens, and appropriate white space.

This is what "load" essentially does - it takes a string and returns a concrete syntax tree [1]. "do" then takes that CST and evaluates it. You can do that in a single-step manner with "do/next" (there's also "load/next"). Moreso, you can think of blocks as phrase markers (see wiki example):

  [S [NP John] [VP [V hit] [NP the [N ball]]]]

Now compare it to Red internal data structure written down in similar fashion:

    ]>

Here I loosely follow phrase marker notation - nonterminals correspond to datatypes, terminals correspond to literal values. But datatype is implied by literal form, and implicitly contained in each value slot (its a datatype ID tag in the header), so I can tidy this up to:

  [1 + 2]

This is what lexer returns - just a block. You can then manipulate it whichever way you want and evaluate.

> at this point you guys don't have a good stepping debugger

We don't have it for a reason - implementing it in user-space would be extremely limited as you can't really distinguish between code and data, and so can't adequately place debugging hooks. The best alternative is just to spruce up block with debugging "print"s and such, or, in case of syntactic errors use "load/trap" and search for error values. Rebol had some interesting projects in this regard, e.g. Anamonitor comes to mind [2]. I think someone already ported it to Red, or re-implemented using our reactive framework.

There was a recent discussion in community chat about that, and we came to conclusion that debugger should rather be implemented at the level of "load" and "do" themselves (e.g. "load" can provide some metainfo, like line numbers and file name, "do" then can single-step it in real time and provide the meaningful info).

[1]: https://en.wikipedia.org/wiki/Parse_tree

[2]: http://rebol2.blogspot.com/2011/11/anamonitor-2-check-block-...