Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Writing a basic x86-64 JIT compiler from scratch in stock Python

csl.name

Post reply on HN
Print view View original

21–30 of 52 posts

Re: Writing a basic x86-64 JIT compiler from scratch in stock Python

#21

Post by weberc2 »

giancarlostoro wrote: post #17
weberc2 wrote: post #16

Earlier quoted context omitted.

I'm writing my first compiler now, and it's been quite a learning curve. I've also realized how much nicer my language is for writing compilers than the language in which I'm implementing the compiler. That made me realize I could write the compiler in my language and then write an interpreter for my language which would only need to run once--to generate the compiler--after which point my compiler would be self-host…

Do you mind sharing some of the resources you're using to learn? Book titles or URLs?

I haven't looked at any resources in particular. Basically my compiler has 3 parts--parser -> type checker -> code generation. Parsing was tricky at first, but then I stumbled upon a pattern which is pretty close to parser-combinators. Type checking seemed daunting, but my language happened to be Hindley-Milner compatible, and there's lots of reference information for HM. Code generation is easy in the base case, but I'm still wrapping my mind around template expansion.

Re: Writing a basic x86-64 JIT compiler from scratch in stock Python

#22

Post by ioddly »

giancarlostoro wrote: post #12
dsco wrote: post #10

Just what I love, a scientific and pedagogic CS article, in well formulated English. I would appreciate some introductory notes or links to prerequisites though. That would perhaps entice an audience who has no previous experience in writing compilers.

I really want to be able to wrap my head around compilers but I'm not sure what a good "user friendly" resource is that wont stuff lots of fancy compiler buzzwords in my face and become white noise to me.

Start simple. If you haven't written an interpreter, do that first. Then figure out how to write a similar program that emits code for a virtual machine rather than evaluating the language.

I'd keep an eye on http://www.craftinginterpreters.com/contents.html, although it hasn't gotten to compilation yet (I think).

Re: Writing a basic x86-64 JIT compiler from scratch in stock Python

#23

Post by blt »

one_thawt wrote: post #13

In my opinion, the most tedious part of writing this sort of thing from scratch is the x86/x86-64 instruction encoding. Grab the Intel/AMD manuals or an opcode database and have fun. Other projects to study: LLVM, dynasm, asmjit, peachpy, luajit jit, Intel XED, etc.

yeah, it sucks. That is what turned me off of initial messing around with JITs. Wish I had known about GNU Lightning then.

Re: Writing a basic x86-64 JIT compiler from scratch in stock Python

#24

Post by OxO4 »

tamer_c wrote: post #6

off topic question: When do you find time do such interesting things as these? I'd really like to dive deeper in this topic. As a CS student, who also got to work as a dev at the same time for a living, I'm even happy to get to sleep 8 hours.

Why not take a class about compilers and implement something like that as a class project?

Re: Writing a basic x86-64 JIT compiler from scratch in stock Python

#25

Post by shakna »

csl wrote: post #19
shakna wrote: post #14

Earlier quoted context omitted.

Lisp In Small Pieces would be my reccomendation. A relatively small book that takes you through writing 11 interpreters and 2 compilers.

Is it really that friendly when starting out, though? I found it brilliant, but quite the challenging read. I'd rather just read chapters four and five of SICP [1] and move on to Queinnec as a follow-up on more advanced topics. [1]: The book is free as well: https://mitpress.mit.edu/sicp/full-text/book/book-Z-H-4.html...

To be honest, I found LISP to be a lot easier than SICP, but that's probably just some learning styles stuff at play.

SICP being freely available in both book form, and the lectures [0] makes it one of the best resources out there.

(I just never found it's section on compilers that helpful. Again, probably learning styles.)

[0] https://www.youtube.com/playlist?list=PLB63C06FAF154F047

Re: Writing a basic x86-64 JIT compiler from scratch in stock Python

#26

Post by Jasper_ »

To be clear about the `del block` thing, the only thing that the `del` keyword does in Python is unbind a name from the local scope. It does not cause an object to be destructed, call `__del__`, or anything else. It's morally equivalent to "block = None", except future references to block will raise a NameError instead of giving you None.

When used at the end of a function, when all locals go out of scope, it is actually doing nothing.

Re: Writing a basic x86-64 JIT compiler from scratch in stock Python

#28

Post by ben-schaaf »

Jasper_ wrote: post #26

To be clear about the `del block` thing, the only thing that the `del` keyword does in Python is unbind a name from the local scope. It does not cause an object to be destructed, call `__del__`, or anything else. It's morally equivalent to "block = None", except future references to block will raise a NameError instead of giving you None. When used at the end of a function, when all locals go out of scope, it is actu…

Unbinding a name does do one thing which might call `__del__`: It decreases the reference count by 1.

Re: Writing a basic x86-64 JIT compiler from scratch in stock Python

#29

Post by tekknolagi »

giancarlostoro wrote: post #12
dsco wrote: post #10

Just what I love, a scientific and pedagogic CS article, in well formulated English. I would appreciate some introductory notes or links to prerequisites though. That would perhaps entice an audience who has no previous experience in writing compilers.

I really want to be able to wrap my head around compilers but I'm not sure what a good "user friendly" resource is that wont stuff lots of fancy compiler buzzwords in my face and become white noise to me.

I'm actually trying to write an easy-to-digest compiler and companion reading series. It's for a compilers independent study class. You can track my progress here: https://github.com/tekknolagi/tigress

At this point it's not well-documented, but the layers are numbered in order: l00, l01, l02... Each builds on top of the previous one.

EDIT: I just added a mini evaluator for debugging purposes and so people can see what the language is like.

Re: Writing a basic x86-64 JIT compiler from scratch in stock Python

#30

Post by dboreham »

OxO4 wrote: post #24
tamer_c wrote: post #6

off topic question: When do you find time do such interesting things as these? I'd really like to dive deeper in this topic. As a CS student, who also got to work as a dev at the same time for a living, I'm even happy to get to sleep 8 hours.

Why not take a class about compilers and implement something like that as a class project?

Where's the fun in that.
Post reply on HN