Live data from Hacker News

Dear sir, you have built a compiler

rachitnigam.com

31–40 of 177 posts

Re: Dear sir, you have built a compiler

#31
I feel like you can apply the same sentiment to many of the "big scary things" in programming.

Things that you don't want to build (as far as "common engineering wisdom" is to be believed):

- a compiler

- a programming language (not sure that there is a difference to compiler as stated in the article)

- a database (query engine)

- a CMS

- a ERP

But sometimes you actually _do_ want to build that (even if every alarm bell in your engineering lizard brain goes off), and then it's probably better to commit and learn the ins and outs of that specific problem domain.

I think especially the recommendation to commit to it is something that's missing from the article. It's "easy" to point a people and saying "look, you did the thing you said you wouldn't do", but far harder to suggest a course of action.

I personally hit the barrier of "things you shouldn't build" in the past, and I've always been happiest (and professional outcomes have been the best) when I decided to break through the barrier and prepare for what lies on the other side, instead of trying to dance around it for ages.

Re: Dear sir, you have built a compiler

#32

I'm always surprised by the lack of "easy" parser/compiler toolkit for more or less complete DSL... It looks like there's only - either full blown programming language for IT guys - "natural language" AI toolkits (not really usable yet for just simple automation by users) - graphical language like State Machine or "no code", missing loops and requiring mouse and boxes However, most of the time, user simply need some…

The easy options:

As others have said, embed something existing. I think this is especially true for programming languages. Even if Lua isn't quite what you want, it's probably going to deliver a lot more value than anything you could bash together on your own. And there are a few other options as well. There are several of these "complete toolkits for DSLs", it's just they do take a bit of work to correctly insert into your code base and then expose enough of what you're doing to make it useful. There isn't and as far as I can see can't be something you just install and it magically exposes everything and it's just awesome and takes no work, because even if you're in something like Python where you can just open an interpreter shell in the current context, thereby technically giving the user the power to do anything they want, you still need to create a safe interface for them, document it, test it, etc. (You may not do a lot of technical work to keep them out of where you haven't tested but you will need to provide them a supported area to get anywhere.)

For building your own: For small tasks, a parser is just a specialized sort of deserializer. Yea verily these 30 years ago, there wasn't a lot of generic deserializers available, and the ones that existed like asn.1 were completely unsuitable for human use, so "classic" compiler literature spends a lot of time on building your own parser. Now, if you don't have a specialized need, you should use a generic deserializer. I have quite a few JSON-based syntaxes lying around. (They feed "interpreters", but same principle.) Lisp S-expressions, if suitable for your environment, can be adapted. XML could be useful in some specialized circumstances where your input looks more like a document than conventional code. Etc. You should only build a custom parser if it's going to give you more bang for your buck across the whole usage cycle than a generic parser can give you. The reason why I have several things that input JSON is precisely that the amount of usage across all time was just going to be too low to justify doing anything specialized, because it was always a strictly internal tool.

(Hint: Just because your compiler accepts JSON as input doesn't mean that that's what the humans have to directly output or edit. My biggest interpreter took in JSON but I provided abundant function helpers that abstracted that away. It was a thin enough abstraction that it could be penetrated if need be, but it avoided most of the verbosity issues as far as the humans were concerned, and gave them the full power of the programming language to generate loops and such. If you squint, I basically borrowed an existing general-purpose programming language that everyone involved already knew to use as a macro language for my language, basically for free. This means the "core" language that I was implementing didn't have to do anything that the already-existing general-purpose language could do in macros, meaning the layer the humans used offered a lot of power for not much implementation cost on my side.)

For AST transforms, this is always where your secret sauce is and there's little help to give, beyond a suggestion that this is arguably the canonical case for a functional programming style, even if you are in an otherwise OO language.

For the final output, I would suggest that in modern times, the easy answer is, don't bother. You can probably interpret fast enough to solve your problems. If not, there are some options like LLVM that will help vs. what you used to have to do. But I would definitely consider just interpreting the final AST directly. If you intend to do that your AST transforms can be built with interpretation in mind.

The real value of a compiler and what distinguishes it from any ol' normal code that ingests something and emits something is the AST in the middle, and the complexity of that AST (probably through significant recursion) and the transforms you run on it, and the corresponding complex behaviors the input can invoke through the recursive input and complex AST transforms. Anything you can do to brush away the incidental details of a compiler by using existing serializations and if at all possible simply skipping the output step of the compiler will hugely increase the bang-for-the-buck you get. You might say, rather than "writing a compiler" consider "using compiler techniques" but not writing a compiler. But not in a way the article was talking about.

For the specific case mentioned there, go get an existing parser for your language, get the full AST tree. You can write your transform in terms of what you do understand and just pass through anything you don't. It's slightly harder, but will be fairly effective.

It is possible even today, of course, that you will be backed against a wall and be forced to go through the full classical compiler cycle. A programming language, for instance, just can't afford to use any existing parser, it's too much a vital part of the experience to try to use an existing one. Even a new Lisp will probably want its own spin on S-expressions enough that it can't just pick up an existing one unchanged. And maybe you've got something that you expect will be big enough that it just requires a new syntax, and you can afford all the support that goes with that. But the vast majority of the time, with the right toolset, you can indeed obtain most of the value of compiler-like techniques without the cost of the full classical compiler technology stack, because even if it was hypothetically possible to build a "real compiler" that had its own custom syntax rather than some slightly dodgy JSON and emitted LLVM code that resulted in an executable that runs all its tasks in .0003 seconds, the engineering effort for that is so much greater than the JSON input and interpreter that runs in .003 seconds that in most cases it's not worth it.

Re: Dear sir, you have built a compiler

#33
post #12

I think you should almost always build a compiler when presented with a parse then execute problem. It is so much faster to create a small machine that is fed data to specific the program, and a test harness that also drives that machine, then it is to hand code every little case repetitively. Or maybe I like building little compilers.

[deleted]

Re: Dear sir, you have built a compiler

#34
You need a compiler when you need language, that is, when you need to interface directly between brain and computer. Everything else can be less messily-specified. A good middle ground is the DSL, domain-specific language. You use the syntax and semantics of the language being used to code in, and change the nouns and verbs. All the nouns and verbs are given meaning as values or functions. Execute in a sandbox so that your DSL can't find the rest of the language. Ruby makes this all trivial.

Of course, if you're trapped in a static language with no way to execute significant logic at runtime without a ton of heavy lifting, then yeah, you'll end up maintaining an interpreter / compiler if you're not careful. Really do yourself a favor and see if you can't introduce Ruby, you can adapt it to your needs. You can inherit from BasicObject, BasicObjects can't see anything you don't want them to see.

Re: Dear sir, you have built a compiler

#35

Related: https://steve-yegge.blogspot.com/2007/06/rich-programmer-foo...

Overall decent take on why you should go for a compilers course, but it has a very weird section advocating for non-deterministic type checking. You want probabilistic AI methods to guess that this string is more of an integer? Buddy, knowing what kind of data we have for certain is the only reason we built type checkers in the first place. Take your 90% chance int but 10% chance string and get out of here.

Maybe he means “error messages are bad, we should guess what people mean and make better suggestions on type errors” but the case was not argued well. It also stuck with an article-long joke of accusing compiler people of being boring at parties. This is 2007 after all, an unenlightened time.

Re: Dear sir, you have built a compiler

#36
The article is “addressed to those who did not want to build a compiler”: what's the point of this article then? It seems to implore to not build compilers, but those who did not want to in the first place, probably actually did not build compilers.

The article does point out numerous challenges with building compilers, and, by extension, with software language engineering, which concerns itself not just with building compilers but with the design of languages, and implementation of non-compiler tools around it. It merely points out the existence, and likely occurrence, of those challenges, and it can be surmised that the author is frustrated by frequent experience in his own daily life.

A bespoke software language -or DSL, if you will- is not always the best solution. It really depends on what you carve out as the domain, how fast that domain changes, what costs and risks are associated with those changes, and with which people you have to implement those changes. No amount of tooling is going to help you out if the domain you chose to recognize as such isn't changing fast enough, isn't costly or risky enough to change (quickly), or the domain stakeholders are anyway not helping out.

But in case you do want to build a DSL, have a look at a language workbench like JetBrains' MPS, or this book I happen to be writing: https://www.manning.com/books/domain-specific-languages-made...

Re: Dear sir, you have built a compiler

#38
post #31

I feel like you can apply the same sentiment to many of the "big scary things" in programming. Things that you don't want to build (as far as "common engineering wisdom" is to be believed): - a compiler - a programming language (not sure that there is a difference to compiler as stated in the article) - a database (query engine) - a CMS - a ERP But sometimes you actually _do_ want to build that (even if every alarm b…

Implementing a small domain-specific language is fairly easy. Why avoid it?

Re: Dear sir, you have built a compiler

#39
Ways you can avoid building a compiler:

1. Lower the scope of the solution. If your solution is designed to address 100% of the users' problems, it's probably too grand of a solution. Start with 80% of the problems. Identify all the problems, rank them by priority and difficulty, and leave the most difficult and least priority problems out.

2. Lower your expectations. Imagine your great idea. Now imagine how you will implement it. Now imagine it is 100x more difficult than you imagine. Woof, that's hard! Strip down the implementation to the essentials needed to solve the immediate problems.

3. Give the user the minimum possible functionality to address their needs.

4. Go find an existing solution that provides these requirements.

If you did it right, you will probably find a solution that you really don't like but already exists and solves the problems you need to solve right now.

Re: Dear sir, you have built a compiler

#40
post #31

I feel like you can apply the same sentiment to many of the "big scary things" in programming. Things that you don't want to build (as far as "common engineering wisdom" is to be believed): - a compiler - a programming language (not sure that there is a difference to compiler as stated in the article) - a database (query engine) - a CMS - a ERP But sometimes you actually _do_ want to build that (even if every alarm b…

I would add encryption/authentication protocols to the list.
Post reply on HN