Live data from Hacker News

My Most Important Project Was a Bytecode Interpreter

gpfault.net

41–50 of 154 posts

Re: My Most Important Project Was a Bytecode Interpreter

#41
post #26
post #22

Earlier quoted context omitted.

The first step in compilation is lexing -- converting a character stream to a stream of semantic "tokens", where a token might be "a number literal" or "the 'while' keyword" or a single character token like "{". This process is usually done via regexs.

A lot of optimizations on arbitrary byte code often look for patterns in byte streams (or in reified assembly) in similar ways to regexes.

For many years, an important stage of the GHC Haskell compiler consisted of a giant Perl script full of regexes.

http://code.haskell.org/ghc-scp/ghc/docs/comm/the-beast/mang...

Re: My Most Important Project Was a Bytecode Interpreter

#42
post #18

Also: a software rasterizer. Most people refuse to write one because it's so easy not to. Why bother? It will make you a better coder for the rest of your life. Let's make a list of "power projects" like this. A bytecode interpreter, a software rasterizer... What else?

Strong agree on emulator and particularly stack vm. Would add: TCP/IP stack.

I once corresponded with someone who wrote his own working TCP/IP stack for his own home PC, an IBM PC Jr.

See Mike's PCJr page near end of this post:

http://jugad2.blogspot.in/2012/09/lissajous-hippo.html

Re: My Most Important Project Was a Bytecode Interpreter

#43
post #32
post #28

Earlier quoted context omitted.

mostly curiosity: why a stack vm in particular? (is it because you then need to write the compiler for it?)

Because if you're familiar with conventional real register architectures, the stack VM forces you to rethink it, in a much simpler way. It's a useful simplifying abstraction.

There used to be a lot of interest in, and articles about, stack machines (real ones, not VMs) back in the days of mags like BYTE and PC Mag. Good reading.

Re: My Most Important Project Was a Bytecode Interpreter

#44
post #4

The project that affected my thinking the most was a bytecode interpreter[1]. I've had use for that knowledge, nearly fifteen years later - most of the interesting learnings about building one has been about the inner loop. The way you build a good interpreter is upside-down in tech - the system which is simpler often works faster than anything more complicated. Because of working on that, then writing my final paper…

Interesting. One thing that I still haven't solved yet is the "break" and "continue" statement inside loops. For a break statement, it seems like it would just be the same a JMP with an address as the operand, but there would need to be some sort of registration of "The VM is in the loop now, and the break address is X", and continue would also be a JMP with an address to the top of the code for the loop.

I haven't implemented those in my system yet, and also have no idea how Python or PHP does it.

Is PHP's VM a stack based one? I do read the Zend/ directory of PHP's source, but it is really hard to follow and there is virtually no documentation on the VM

Re: My Most Important Project Was a Bytecode Interpreter

#46
post #42
post #18

Earlier quoted context omitted.

Strong agree on emulator and particularly stack vm. Would add: TCP/IP stack.

I once corresponded with someone who wrote his own working TCP/IP stack for his own home PC, an IBM PC Jr. See Mike's PCJr page near end of this post: http://jugad2.blogspot.in/2012/09/lissajous-hippo.html

Update: Visited his page again. He has now even created a web server for the PCJr and ran his site on it for a while.

Re: My Most Important Project Was a Bytecode Interpreter

#47
post #27

Two approaches are severely underused in the software world: 1) Domain-specific languages (DSLs) 2) Virtual machines (or just explicit state machines more generally) What I mean is, alot of problems could be solved cleanly, elegantly, more safely, and more powerfully by using one (or both) of the above. The problem is that when people think DSL or VM, they think big (Scheme or JVM) instead of thinking small (printf).…

Yes, can you describe your SPF library? I'd love to learn more about how using a VM / state machine paradigm could help me approach CS problems.

You can find it here:

  https://github.com/wahern/dns/blob/master/src/spf.rl
(The Ragel stuff is for parsing the SPF records and is a rather small detail. I always intended to remove it--it's a dependency that turns alot of people off--but never got around to it. But, FWIW, Ragel is awesome when you don't mind the build-time dependency.)

Basically what happened was that when I set out to write an async SPF library I found myself constantly refactoring the internal API and devising ever more clever hacks for implementing continuations in C. The biggest problems were

1) It had to support asynchronous I/O, and specifically non-blocking I/O.

2) I explicitly DID NOT want to use callbacks because one design goal of the library was that it be easy to integrate into other code and frameworks, regardless of event loop and even without an event loop. I had recently implemented a non-blocking DNS library after years of using and hacking on ADNS and c-ares taught me to hate callbacks in C.

Instead, you repeatedly call spf_check() until you get something other than EAGAIN. When you get EAGAIN, you use spf_pollfd(), spf_events(), and spf_timeout() for the polling parameters. (The file descriptor can change dynamically, e.g. if the DNS query had to switch to TCP from UDP. The events can be POLLIN or POLLOUT, though usually POLLIN, and the timeout is necessary because DNS send/recv timeouts can be much smaller than the timeout for the whole SPF check operation.)

3) SPF is recursive--policies can include other policies, which can include other policies. So you're already dealing with a logical stack situation, but you can't rely on the language's runtime stack. Also, for each rule there can be an iteration over DNS records with intermediate lookups, and it's especially ugly in C to pause and resume loops with inner function calls.

4) I had never implemented SPF before. I kept running into areas where my approach turned out to be deficient as I became more familiar with the spec. And because of constraints 1-3, this was very costly in time and effort. Were I to implement an SPF library again I probably wouldn't use a VM; instead I'd go back to using a basic state machine, function jump table, and a simpler state stack. But at the time it made sense because it was a more flexible and conceptually simpler approach given the unknowns. Plus, it was an excuse to try out a concept I had long had, which was implement a VM for non-blocking I/O operations.

So basically what it does is decompose the terms in an SPF policy to logical opcodes. So the "ip4:1.2.3.4" term composes into an opcode which pushes a typed address (1.2.3.4) onto the stack and the opcode (OP_IP4) for matching the address at the top of the stack to the address of the connecting host (which is part of the context of an instantiated SPF processor object).

The term "mx:foo.bar", for example, requires querying MX records for foo.bar, iterating over each host and recursively querying the A or AAAA records, and then matching the IP address like above. The "mx:" term is first emitted like the "ip:" term--push domain string following my OP_MX. But when the OP_MX opcode is executed it dynamically generates more bytecode to do the query. I don't remember why I did it this way--probably because it seemed simpler/easier at the time.

It works well, but objectively is over-engineered. At the very least the VM is _too_ general. Nonetheless, AFAIK it's the most comprehensive async I/O SPF library in C that doesn't use callbacks. It only passes ~80% of the SPF test suite, but that's mostly because of the policy parser--IIRC the ABNF grammar for SPF is broken because the proof-of-concept was implemented using a naive regular expression to chop of the string (the ABNF translation of the pattern is ambiguous); and I never bothered or needed to handle some of the corner cases necessary to match the behavior of, e.g., libspf2. But my implementation handles many millions (if not billions) of queries every day as it's used in several commercial and cloud products.

Re: My Most Important Project Was a Bytecode Interpreter

#48

This is a required hw assignment for a freshmen class @ cmu. https://www.cs.cmu.edu/~fp/courses/15122-s11/lectures/23-c0v... Given it has some parts already written in the interest of time...

I am jealous of those students. I would love to take that class.

Re: My Most Important Project Was a Bytecode Interpreter

#49

Also: a software rasterizer. Most people refuse to write one because it's so easy not to. Why bother? It will make you a better coder for the rest of your life. Let's make a list of "power projects" like this. A bytecode interpreter, a software rasterizer... What else?

Hmm, rasterizer was just a starting point for me on the way to understand "how browser works".

Having just a rasterizer is like naked VM without bytecode compiler.

So I decided to add HTML/CSS engines to it.

But if you have HTML/CSS then why not to add scripting to it? So added VM executing bytecodes, GC and compiler producing those bytecodes. Having VM I thought that it would be cool to have built-in persistence in it - you need to persist UI state somehow, right? So it got integrated NoSQL database on board.

That's pretty much how http://sciter.com was born.

Re: My Most Important Project Was a Bytecode Interpreter

#50
post #9

Earlier quoted context omitted.

I really recommend a raytracer, especially to anyone interested in graphics. It's straightforward, powerful, infinitely expandable with optional features, and opens up a ton of discussion about performance, code complexity, and general organization. Plus it's fun, in an instant gratification kind of way.

Every now and then I get interested in demoscene programming. I've never even been able to get a triangle to render on the screen - except with something like XNA. Do you think there's any value in going back to say, DOS based VGA programming? People in #osdev thought I was a bit strange for wanting to write a bootable kernel that only put pixels on the screen, but I really enjoy the idea of starting with plotting pi…

You could skip writing a bootable kernel, and try that with microcontrollers :) It's possible to hook up a tiny lcd screen to an arduino
Post reply on HN