Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Dear sir, you have built a compiler

rachitnigam.com

Post reply on HN
Print view View original

111–120 of 177 posts

Re: Dear sir, you have built a compiler

#111

Post by BeetleB »

hobofan wrote: post #31

I feel like you can apply the same sentiment to many of the "big scary things" in programming. Things that you don't want to build (as far as "common engineering wisdom" is to be believed): - a compiler - a programming language (not sure that there is a difference to compiler as stated in the article) - a database (query engine) - a CMS - a ERP But sometimes you actually _do_ want to build that (even if every alarm b…

I had a very different reading of the article, colored by my own bad experience.

Sometimes you start off with this cool idea/hack to solve a problem that could have been solved some other way. But your cool hack solves the problem and more! Then your fellow team members all want it to be even more powerful, and your cool hack slowly grows into a monstrosity.

In these situations, often your manager did not particularly want you to do this, but he let you do what you want. He does want you to be productive, and this doesn't count for that much. But since the team is now dependent on you, he does expect you to maintain it. You now end up with two jobs: The one your manager hired you for, and maintaining this thing.

I would love to write my own DSL to solve a problem at work, and this post read to me as a cautionary tale on pursuing that goal - unless management is aligned with your vision. If they're not, solve it the boring (and perhaps nonideal) way.

Re: Dear sir, you have built a compiler

#112

Post by nybble41 »

jasoncabot wrote: post #56

This reminds me of a great post about the typical software evolution going from hard coded values, through configuration and rules engines to a DSL, only to be back where you started. I have seen it happen on multiple projects, it’s hard to spot it at the time as all the changes seem sensible but hindsight is a wonderful thing http://mikehadlow.blogspot.com/2012/05/configuration-complex...

IMHO an embedded DSL (eDSL) helps to close the loop somewhat. At least you're writing your "configuration code" in a real programming language, while still keeping the configuration separate from the main application. It addresses most of the issues the article points out regarding custom DSLs: no/limited debug capability, lack of tooling and IDE support, unable to write the same style of code used in the application. Of course, it helps if your main language has good support for eDSLs, since not all do.

Re: Dear sir, you have built a compiler

#113

Post by mst »

wpietri wrote: post #78
jasoncabot wrote: post #56

This reminds me of a great post about the typical software evolution going from hard coded values, through configuration and rules engines to a DSL, only to be back where you started. I have seen it happen on multiple projects, it’s hard to spot it at the time as all the changes seem sensible but hindsight is a wonderful thing http://mikehadlow.blogspot.com/2012/05/configuration-complex...

This line really struck me: “They will have to change at some point, and you don’t want to recompile and redeploy your application just to change the VAT tax rate.” Continuous deployment has changed this for me. I'm still not going to leave tunable constants in the middle of random if statements. But nowadays instead of trying to avoid recompiling and redeploying, I'd rather invest that time into making that very eas…

Pulling things out into a configuration file (or environment variable) is valuable when either (a) the value needs to be set per-environment and you have a variable number of environments or (b) the value needs to be changeable by somebody other than the dev team.

Also it's worth considering for constants whether what you actually want is a 'resource file' (which could easily just be a file containing all your constants written in your programming language, no need to use a different syntax if that doesn't gain you anything) - learning and internalising (and also learning how to explain to other developers) that a resource file is -not- the same as a configuration file has been a significant quality of life improvement for me.

Re: Dear sir, you have built a compiler

#114

Post by hinkley »

sunir wrote: post #12

I think you should almost always build a compiler when presented with a parse then execute problem. It is so much faster to create a small machine that is fed data to specific the program, and a test harness that also drives that machine, then it is to hand code every little case repetitively. Or maybe I like building little compilers.

Parser yes, compiler no.

It occurred to me at several points in my career that the tools I was using were deeply hamstrung by mixing interpretation of complex data with the use of that data, making it difficult for anyone to effectively add new behavior in the system. Most vividly, I ran into this with first Trac (which supports plugins, but at the time each one had to reimplement scanning of wiki words and all handled a different 80% of all scenarios) and Angular 1.0.

A variant on this, I don't know that I've ever heard a coined phrase for but I just call Plan and Execute. In the spirit of dynamic programming, you visit a graph of data once in order to enumerate all of the tasks you will need to do, deduplicate the tasks, and then fire them off. This works much better than caching because under load a single unit of work can potentially self-evict. It's also a lot easier to test. It does however delay the start of work, so either you need to make that up in reduced runtime (by not doing things twice) or have a mixed workload. You may also have to throttle outbound requests that had previously been staggered by alternating CPU and IO bound phases.

I have one bit of production code where this is working well, and another that desperately needs it but is challenging to reach feature parity due to the way it is invoked and limitations of a library that it uses. I'm going to have to change one or both.

Re: Dear sir, you have built a compiler

#115

Post by cdcarter »

agumonkey wrote: post #100
chubot wrote: post #79

Earlier quoted context omitted.

IIRC it's even in the "Design Patterns" book -- the Interpreter pattern. It really does come up a lot.

The GoF book ? .. I didn't remember seeing it but I have to admit I skimmed with very negative eyes :)

Yep, the interpreter pattern is definitely in GoF. Good way to learn a basic stack interpreter.

Re: Dear sir, you have built a compiler

#116

Post by cjbgkagh »

hobofan wrote: post #31

I feel like you can apply the same sentiment to many of the "big scary things" in programming. Things that you don't want to build (as far as "common engineering wisdom" is to be believed): - a compiler - a programming language (not sure that there is a difference to compiler as stated in the article) - a database (query engine) - a CMS - a ERP But sometimes you actually _do_ want to build that (even if every alarm b…

Lol, I’ve built all those things.

I think the main thing is having respect for the mountain of engineering that went into existing solutions and not rolling your own unless you really really have to.

I think an Embedded DSL with post compilation constraint solver based analyzers could take the place of restricted mini-languages while taking advantage of the existing engineering and tooling available for modern languages.

Re: Dear sir, you have built a compiler

#117

Post by cmrdporcupine »

hobofan wrote: post #31

I feel like you can apply the same sentiment to many of the "big scary things" in programming. Things that you don't want to build (as far as "common engineering wisdom" is to be believed): - a compiler - a programming language (not sure that there is a difference to compiler as stated in the article) - a database (query engine) - a CMS - a ERP But sometimes you actually _do_ want to build that (even if every alarm b…

I mean I actually really want to build many of those things (the first 3 anyways), and I have. I just don't generally show the results to people or make them use them.

I can't vouch for the quality of what I've built, but taking time to attempt to build those kinds of things has always paid off in terms of overall betterment as a programmer.

Then I can go back to a day job of mundanely moving protobufs from one place to another.

Re: Dear sir, you have built a compiler

#118

Post by DoctorDabadedoo »

jimmaswell wrote: post #38
hobofan wrote: post #31

I feel like you can apply the same sentiment to many of the "big scary things" in programming. Things that you don't want to build (as far as "common engineering wisdom" is to be believed): - a compiler - a programming language (not sure that there is a difference to compiler as stated in the article) - a database (query engine) - a CMS - a ERP But sometimes you actually _do_ want to build that (even if every alarm b…

Implementing a small domain-specific language is fairly easy. Why avoid it?

Because if it succedds, it needs to be supported long term. If you're doing that inhouse, chances are slim of original designers staying 10-20 years at the same company.

Then what you have in your hands is a legacy monolith neglected over its life which might double as core of your stack and it's awful to move away from down the line.

Been there, don't recommend it.

Re: Dear sir, you have built a compiler

#119

Post by chillingeffect »

hobofan wrote: post #31

I feel like you can apply the same sentiment to many of the "big scary things" in programming. Things that you don't want to build (as far as "common engineering wisdom" is to be believed): - a compiler - a programming language (not sure that there is a difference to compiler as stated in the article) - a database (query engine) - a CMS - a ERP But sometimes you actually _do_ want to build that (even if every alarm b…

- a filesystem

Re: Dear sir, you have built a compiler

#120

Post by munificent »

hobofan wrote: post #31

I feel like you can apply the same sentiment to many of the "big scary things" in programming. Things that you don't want to build (as far as "common engineering wisdom" is to be believed): - a compiler - a programming language (not sure that there is a difference to compiler as stated in the article) - a database (query engine) - a CMS - a ERP But sometimes you actually _do_ want to build that (even if every alarm b…

I would also add a build system to that list, a tarpit many engineers have fallen into.

> not sure that there is a difference to compiler as stated in the article

It's hugely different.

If you're building a compiler for an existing language, you inadvertently signed yourself up for implementing and supporting the entire language, which is a pretty large engineering task.

But if you design a new language, you signed up for not just implementing the language, but also:

- Actually designing it, which is a very difficult task with hard trade-offs all over the place.

- Documenting it so that others can actually learn it. Technical writing is another hard task that takes years to master.

- Syntax highlighting and editor support so that it's as easy to work with as other languages. You will discover that users in the wild apparently use thousands of different editors, all of which you are expected to support.

- Static analysis tooling so that code navigation, automated refactoring, linting, etc. all work. Users today expect a rich IDE experience for every language they use. Also, you'll discover the hard way that the architecture of an interactive static analyzer is very different from your batch mode compiler, so now you're likely writing two front ends.

- Thousands of answers on StackOverflow, blog posts, introductory videos, etc. Users learn in many different ways and they expect to find education material in the form they prefer.

Post reply on HN