Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Writing a basic x86-64 JIT compiler from scratch in stock Python

csl.name

Post reply on HN
Print view View original

31–40 of 52 posts

Re: Writing a basic x86-64 JIT compiler from scratch in stock Python

#31

Post by ATsch »

ben-schaaf wrote: post #28
Jasper_ wrote: post #26

To be clear about the `del block` thing, the only thing that the `del` keyword does in Python is unbind a name from the local scope. It does not cause an object to be destructed, call `__del__`, or anything else. It's morally equivalent to "block = None", except future references to block will raise a NameError instead of giving you None. When used at the end of a function, when all locals go out of scope, it is actu…

Unbinding a name does do one thing which might call `__del__`: It decreases the reference count by 1.

In CPython, yes. But this is an implementation detail.

Re: Writing a basic x86-64 JIT compiler from scratch in stock Python

#32

Post by userbinator »

giancarlostoro wrote: post #12
dsco wrote: post #10

Just what I love, a scientific and pedagogic CS article, in well formulated English. I would appreciate some introductory notes or links to prerequisites though. That would perhaps entice an audience who has no previous experience in writing compilers.

I really want to be able to wrap my head around compilers but I'm not sure what a good "user friendly" resource is that wont stuff lots of fancy compiler buzzwords in my face and become white noise to me.

that wont stuff lots of fancy compiler buzzwords in my face and become white noise to me.

Crenshaw's tutorial fits this perfectly: https://compilers.iecc.com/crenshaw/

x86 version here: http://www.pp4s.co.uk/main/tu-trans-comp-jc-intro.html

Re: Writing a basic x86-64 JIT compiler from scratch in stock Python

#33

Post by lboasso »

userbinator wrote: post #32
giancarlostoro wrote: post #12

Earlier quoted context omitted.

I really want to be able to wrap my head around compilers but I'm not sure what a good "user friendly" resource is that wont stuff lots of fancy compiler buzzwords in my face and become white noise to me.

that wont stuff lots of fancy compiler buzzwords in my face and become white noise to me. Crenshaw's tutorial fits this perfectly: https://compilers.iecc.com/crenshaw/ x86 version here: http://www.pp4s.co.uk/main/tu-trans-comp-jc-intro.html

I think Wirth's compiler book is more approachable for a beginner. Once you are done with that book you can also read "Project Oberon" from the same author. This way you can understand the implementation of a full graphical OS written from scratch for a custom made RISC processor implemented on a FPGA. The book and source code are available at: http://www-oldurls.inf.ethz.ch/personal/wirth/ProjectOberon/

Re: Writing a basic x86-64 JIT compiler from scratch in stock Python

#34

Post by eliben »

one_thawt wrote: post #13

In my opinion, the most tedious part of writing this sort of thing from scratch is the x86/x86-64 instruction encoding. Grab the Intel/AMD manuals or an opcode database and have fun. Other projects to study: LLVM, dynasm, asmjit, peachpy, luajit jit, Intel XED, etc.

FWIW, the series concluding at https://eli.thegreenplace.net/2017/adventures-in-jit-compila... build a Brainf*ck JIT in pure C++, then using LLVM, then in Python using PeachPy

Re: Writing a basic x86-64 JIT compiler from scratch in stock Python

#35

Post by abecedarius »

giancarlostoro wrote: post #17
weberc2 wrote: post #16

Earlier quoted context omitted.

I'm writing my first compiler now, and it's been quite a learning curve. I've also realized how much nicer my language is for writing compilers than the language in which I'm implementing the compiler. That made me realize I could write the compiler in my language and then write an interpreter for my language which would only need to run once--to generate the compiler--after which point my compiler would be self-host…

Do you mind sharing some of the resources you're using to learn? Book titles or URLs?

I wrote https://codewords.recurse.com/issues/seven/dragon-taming-wit... about a tutorial-level compiler written in itself. Of course, a recommendation from someone who's learned from it instead of from the author would give a better signal.

I also recommended the same Wirth book as lboasso, and a couple other books, down at the bottom of the page.

Re: Writing a basic x86-64 JIT compiler from scratch in stock Python

#37

Post by UncleEntity »

kovrik wrote: post #3

That is awesome! Thanks! Does anybody know the easiest way to compile to JVM bytecode? I have a Scheme interpreter written in Kotlin, what is the best way to compile it, instead of interpreting? Where do I start?

I was poking around* with this yesterday, has two different simple implementation of a lispkit compiler:

http://www.cs.ncl.ac.uk/publications/trs/papers/129.pdf

*by "poking around" I mean I typed the code into a text editor for further study:

code: https://pastebin.com/6pAkqE2R

Re: Writing a basic x86-64 JIT compiler from scratch in stock Python

#38

Post by defined »

This is a very interesting and well-written article, but I am puzzled by the technique being described as "JIT compilation".

Isn't JIT compilation the process of compiling some sort of intermediate code into machine code at runtime? For example, compiling JVM bytecode into machine code during execution, and using the cached machine code transparently on subsequent calls to that segment of code.

Not to detract from the article, or the interesting techniques and explanation, but I didn't see any compilation other than by gcc, which IMHO makes this AOT rather than JIT compilation.

What am I missing?

Re: Writing a basic x86-64 JIT compiler from scratch in stock Python

#39

Post by rdc12 »

defined wrote: post #38

This is a very interesting and well-written article, but I am puzzled by the technique being described as "JIT compilation". Isn't JIT compilation the process of compiling some sort of intermediate code into machine code at runtime ? For example, compiling JVM bytecode into machine code during execution, and using the cached machine code transparently on subsequent calls to that segment of code. Not to detract from t…

For me the key concept that comes to mind for a JIT compiler, is profiling at runtime to decide what to compile and how aggressively which this article doesn't deal with. The other big thing is how to make the profiling and compilation not be too expensive to negate itself.

The article author seems to be using GCC not at runtime, but more as a generator for templates (at the machine code level) for the rest of the software to use. You don't really want to be assembling a template at runtime in this context.

Post reply on HN