Earlier quoted context omitted.
> But if somebody believes that writing a transpiler isn’t fundamentally the same thing as writing a compiler, I'd be surprised if anyone did. The use of transpiler is more about audience expectation, a specificity. It's shorter than writing "source-to-source compiler", and acknowledges compiler as its superset, right there in its name
> It's shorter than writing "source-to-source compiler" Brevity isn't everything. But more importantly, "transpiler" (like "source-to-source compiler") does not say what you are compiling from , and what you are compiling to . In order for the term "transpiler" to be useful, you need to specify those things. Anything you think you imply by using the term is not, in fact, implied. If you compare the lengths of "JavaSc…
My first fifteen compilers
71–77 of 77 posts
Re: My first fifteen compilers
#72Favorite quote: > There’s a wealth of tutorials, courses, books, and the like about how to write compilers. But if somebody believes that writing a transpiler isn’t fundamentally the same thing as writing a compiler, it may not occur to them to look at any of that material. The basic argument is this: "compiler" isn't a term that needs to be limited from transforming a high-level input to a low-level output. Any prog…
You got me thinking. Can a human language like English be defined with an AST? If so, are there examples? If not, why not? I suspect the answer might be, "yes, it's called [this thing I've heard of a thousand times but never considered it to be a language compiler]"
Re: My first fifteen compilers
#73I suppose if you look at it that way, then anyone who has completed Crenshaw's excellent tutorial series[1] could also claim to have written (approximately) the same number of "compilers". With a parser combinator library, you write a parser by starting with a bunch of primitive parsers (say, that parse numbers or characters) and combining them, eventually building up the ability to parse a sophisticated language. Th…
A counter example to this is the typescript compiler which some might describe as a transpiler. It does sophisticated control flow analysis to, for example, make sure all branches of an if statement return the same type.
Re: My first fifteen compilers
#74Earlier quoted context omitted.
> "Transpiler" is short for "transcompiler". Has been since the 80's. Citation needed that either of these terms actually existed before 2013. https://books.google.com/ngrams/graph?content=compiler%2Ctra... https://trends.google.com/trends/explore?date=all&q=transpil... https://trends.google.com/trends/explore?date=all&q=transcom...
> Citation needed that either of these terms actually existed before 2013. Your own sources have references in '03, which contradicts that. But I'll oblige. Sitting on my shelf is "XLT86 - 8080 to 8086 Assembly Language Translator, User Guide", dated 1982. (But not the version you can find online, which is September, '81. I'm not quite sure on the release, as the user guides didn't include a revision number). Here's…
Re: My first fifteen compilers
#75Earlier quoted context omitted.
Lexers split a program (string) into groups of characters as a list of tokens. Lexical analysis has nothing to do with syntax. Canonically, syntax and grammar are the same thing and parsers deal with that. A parser looks for patterns of tokens that match up to a grammar rule. If all the grammar rules that compose a correct program are parsed, a parser produces an AST that the compiler can act on. Otherwise the parser…
The boundaries between lexers and parsers differ by the parser in question. Grammar and syntax are also different, particularly with regards to XML based languages. Syntax are the rules which define the language while grammars are the conventions that define the context in which artifacts in a language instance are interpreted. Whether a language requires terminating semicolons or curly braces are syntax rules. Wheth…
Indeed, the division between lexer and parser is somewhat optional, since scannerless parsers can be defined to operate directly on a stream of basic symbols from the language's alphabet, but the usual division reflects notions from automata theory:
- A lexer (a.k.a., lexical analyzer, tokenizer, scanner) is founded, in principle if not in actuality, on a finite state automaton to group input symbols from an alphabet into basic meaningful units ("lexemes" or "tokens") and to classify those units according to their significance (identifiers, keywords, literals, delimiters, etc.). In a natural language context, it can be thought of as producing words from a sequence of letters.
- A parser is founded, in principle if not in actuality, on (usually and at least) a finite state pushdown automaton, which is basically a finite state automaton equipped with a stack which can be manipulated by the transition function. The goal of a parser is to convert a stream of input symbols into one (or sometimes more) derivations (a.k.a. parse trees, [concrete] syntax trees, parses) which represent the structure of the input in terms of a grammar. In a natural language context, it can be thought of as producing sentence diagrams from a sequence of words (or letters in the case of scannerless parsing).
> Grammar and syntax are also different,
No, grammar and syntax are the same. A grammar consists of an array of productions, which are rules that define how symbols in the language may be combined to form valid sentences in the language. Usually, "grammar" refers to a formalism that is sorta like a big regular expression, except that it's not limited to the regular languages (context-free grammars being most common, with extra-parser hacks to support context-sensitive languages if needed), but a grammar is really an abstract mathematical object and need not be formally manifest.
> Syntax are the rules which define the language while grammars are the conventions that define the context in which artifacts in a language instance are interpreted.
Indeed, syntax can be thought of as the rules which define a language (that is, can be thought of as a grammar). However, what you describe as "the conventions that define the context in which artifacts in a language instance are interpreted" is properly called semantics, which are rules for ascribing meaning to phrases in the language.
> Parsers commonly produce abstract syntax trees (AST), but can produce output in a variety of formats. I prefer parse tables personally. To say that a parser must produce as AST is rather short-sided and inexperienced.
This is true; indeed a parser can directly produce any sort of output: a single truth value that indicates whether the input is part of the language that it recognizes, a translation into another language, or even "no" output in the case of an interpreter. However, I'm confused by your usage of "parse tables" here. The term usually refers to the tables that are used to drive a parsing algorithm (i.e., an input to the parsing algorithm) rather than the output of a parser. Would you elaborate for me?
> Lexers, parsers, and compilers are all separate steps that can act independently provided a sufficient configuration.
Indeed, a parser does not necessarily require a lexer, nor must a parser's output be compiled. However, though your statement is technically true, I can't even imagine what a compiler without a parser might look like!
Re: My first fifteen compilers
#76Anyone got stories about attempts to combine compiler construction with deep learning techniques? As AI related technologies now become realistically implementable, wouldn't the compiler theory be one of the most greatly affected research fields?
Why would it? What techniques do you propose to use for what tasks?
Re: My first fifteen compilers
#77Most compilers sort of work this way, except that they don't have an intermediate format. They take source, turn it into tokens, turn it into a tree of sorts, run several passes over it till the end result is reached (whatever the target is). My own compilers turn 4 different input languages in the same parser tree (Hand written tokenizers/parsers), do several passes over it until it has a very base set of instructio…
It's not that there's a textual intermediate format: just a defined intermediate tree structure. Some examples are given in the nanopass framework documentation [0]. So it's not that the input language is processed into an intermediate format, which is printed to a string and then read by the next pass; the intermediate languages are all in various forms of trees. See also the paper on writing Chez Scheme as a nanopa…