My Most Important Project Was a Bytecode Interpreter

Tracks I've done and suggested to friends and colleagues as learning experiences:

* Compression (lossless, lossy, image, audio, texture, video)

* Languages (bytecode interpreter, AST interpreter, parser/lexer for a simple language, simple JIT, understanding instruction scheduling)

* DSP programming (writing programs for fast, branchless math)

* Comfort with binary and binary formats (start with packfiles .zip/.tar, move onto reverse engineering simple formats for e.g. games)

* Understanding the difference between RAM and address spaces (e.g. understanding virtual memory, mmap, memory-mapped IO, dynamic linking, the VDSO, page faulting, shared memory)

* Device drivers (easier on Linux, understanding userspace/kernel interaction, ioctls, how hardware and registers work, how to read spec sheets and hardware manuals)

* Graphics (modern software rasterizer that's not scanline-based, understanding 3D and projective transforms, GPU programming and shaders, basic lighting (reflection and illumination) models, what "GPU memory" is, what scanout is, how full scenes are accumulated all along the stack)

I could go heavily into depth for any one of these. Ask me questions if you're interested! They're all fun and I always have more to learn.

Also, the longer you get into any given track, the more you realize they all connect in the end.

A web server. Use a subset of HTTP 1.0 and make the browser serve pages.

A chat server. One chat room. Make sure you can handle fast and slow clients, and disconnects.

Two approaches are severely underused in the software world:

1) Domain-specific languages (DSLs)

2) Virtual machines (or just explicit state machines more generally)

What I mean is, alot of problems could be solved cleanly, elegantly, more safely, and more powerfully by using one (or both) of the above. The problem is that when people think DSL or VM, they think big (Scheme or JVM) instead of thinking small (printf). A DSL or VM doesn't need to be complex; it could be incredibly simple but still be immensely more powerful than coding a solution directly in an existing language using its constructs and APIs.

Case in point: the BSD hexdump(1) utility. POSIX defines the od(1) utility for formatting binary data as text, and it takes a long list of complex command-line arguments. The hexdump utility, by contrast, uses a simple DSL to specify how to format output. hexdump can implement almost every conceivable output format of od and then some using its DSL. The DSL is basically printf format specifiers combined with looping declarations.

I got bored one day and decided to implement hexdump as a library (i.e. "one hexdump to rule them all"), with a thin command-line wrapper that emulates the BSD utility version. Unlike BSD hexdump(1) or POSIX od(1), which implement everything in C in the typical manner, I decided to translate the hexdump DSL into bytecode for a simple virtual machine.

  http://25thandclement.com/~william/projects/hexdump.c.html

The end result was that my implementation was about the same size as either of those, but 1) could built as a shared library, command-line utility, or Lua module, 2) is more performant (formats almost 30% faster for the common outputs, thanks to a couple of obvious, easy, single-line optimizations the approach opened up) than either of the others, and 3) is arguably easier to read and hack on.

Granted, my little hexdump utility doesn't have much value. I still tend to rewrite a simple dumper in a couple dozen lines of code for different projects (I'm big on avoiding dependencies), and not many other people use it. But I really liked the experience and the end result. I've used simple DSLs, VMs, and especially explicit state machines many times before and after, but this one was one of the largest and most satisfying.

The only more complex VM I've written was for an asynchronous I/O SPF C library, but that one is more difficult to explain and justify, though I will if pressed.

OMG, memories... at once point in the pleistocene era, I was tasked with writing a manual for what was then Informix 4GL[1], and to get comfortable with the language, I spent a week writing a user forum app, with topics and message threads, messages stored as "blobs" in the SQL DB. Tried to get my co-workers in the pubs group to use it. They thought it was cute but all said hahaha no.

[1] https://en.wikipedia.org/wiki/IBM_Informix-4GL -- I am flabbergasted to see 4GL is still in use 30 years on.