I find it surprising that a single-pass compiler is easier to implement than a traditional lexer->parser->AST->emitter. (I'm not a compiler expert, though.) I'd have expected that generating an AST would be at least as simple, if not simpler. Plus by generating an AST, doing some simple optimization is a lot easier: one can pattern-match parts of the AST and replace them with more efficient equivalents. Maybe I'm ove…
Writing a C compiler in 500 lines of Python (2023)
51–60 of 114 posts
Re: Writing a C compiler in 500 lines of Python (2023)
#52After many years of programming in other languages, I finally learned C, and came to realize that there aren't actually any compilers that implement all of the C spec. Even GCC and Clang have their grey areas and their bugs. Before this, I had thought that C was a simple language. An idea propped up by articles likes this, as well as the oft touted fact that nearly every embedded system has a C compiler; no matter wh…
This blog post is full of misconceptions.
It starts by asserting, that C defines an ABI and then complains that everything is so complicated, because actually C doesn't define it. C is defined in terms of behaviour of the abstract C machine. As far as C is concerned, there is no ABI. C only prescribes meaning to the language, it does not prescribe how you actually implement this; the compiler is free to do as it pleases, including choosing the ABI in some limits.
What defines the ABI is the platform consisting of the machine and the OS. The OS here includes at least the kernel (or some bare metal primitives) and the compiler. And the standard C library really IS part of the compiler. That's why GCC vs Clang or glibc vs muslc always comes with incompatibilities. Because these ARE different OSs. They can choose to do things the same, but this is because of some formal (POSIX, the platform vendor) or some informal (GCC and Clang) standards.
Yes a lot of ABIs are defined with C syntax, but this is, because C has terms for that and isn't too new. You can specify this in the language of your choice and it will describe the same ABI. Yes, int doesn't have a size independent of the platform. But if the specification wouldn't use C as a syntax, it would just write "this argument has the same size as described in the 'Appendix X Definitions' under 'Platform's default integer size'". Writing "int" is just a shorter notation for exactly this.
> You Can’t Actually Parse A C Header
I don't know why the choice to use the compiler to implement parsing a C header is framed as a bad thing. Is relying on write(2) from the kernel a bad thing instead of trying to bypass the kernel? The compiler is what defines the meaning of a header, why don't ask it about the result? If you don't feel like reimplementing the C preprocessor, you can also just parse preprocessed headers. These are self-contained, i.e. don't need knowing the include path. But of course this approach comes with the caveat that when the user updated the C compiler, your knowledge has become outdated or wrong. I don't know why it is framed as weird, that you need a C parser to parse C code. This is the definition of a C parser. You can't just write code that parses C and is somehow not a C parser.
> 176 triples. I was originally going to include them all for the visual gag/impact but it’s literally too many for even that.
No, they are ONLY 176 target triples (this is the LLVM term, other terms are "gnu tuple" or "gnu type") that your tool supports. There is also not the definite list, it's a syntax to describe the major components of a platform. There are decades of vendors improving their platform in incompatible ways, of course the description of this is messy.
See for example: https://news.ycombinator.com/item?id=43698363
And this is the test data for the source of GNU types: https://cgit.git.savannah.gnu.org/cgit/config.git/tree/tests... See that this contains 1180 types, but of course that's also not definite.
> pub type intmax_t = i64;
> A lot of code has completely given up on keeping C in the loop and has started hardcoding the definitions of core types. After all, they’re clearly just part of the platform’s ABI! What are they going to do, change the size of intmax_t!? That’s obviously an ABI-breaking change!
There is a reason it is called intMAX_t! It does not HAVE a definite size, it is the MAXimal size of an integer on that platform. Yes, they are problems nowadays due to ossification, but they come exactly from people like that blog author. When you want your program to have a stable ABI, that doesn't change when your platform supports larger integer types, you just don't use intMAX_t!
> And even then you have the x64 int problem: it’s such a fundamental type, and has been that size for so long, that countless applications may have weird undetectable assumptions about it. This is why int is 32-bit on x64 even though it was “supposed” to be 64-bit: int was 32-bit for so long that it was completely hopeless to update software to the new size even though it was a whole new architecture and target triple!
That is called ossification. When you program C you are not supposed to care about the sizes. When your program does, your program is broken/non-portable. Yes, this limits the compilers, because they don't want programs to be broken. But is really the same as e.g. MS Windows catering to a specific program's bugs. This is not a design mistake of C:
> sometimes you make a mistake so bad that you just don’t get to undo it.
Re: Writing a C compiler in 500 lines of Python (2023)
#53Earlier quoted context omitted.
That is a myth often spread by folks that think K&R C book is everything there is to know, never opened the ISO C draft PDF, learned the differences between POSIX and standard library, tried to make their code portable outside GCC or nowadays clang, or even checked the extensions chapter on the compiler.
Who told that? I still hear K&R being recommended as introduction material. If you want to write portable/production-grade C code, well definitely need to study another references.
Re: Writing a C compiler in 500 lines of Python (2023)
#54Earlier quoted context omitted.
Who told that? I still hear K&R being recommended as introduction material. If you want to write portable/production-grade C code, well definitely need to study another references.
Exactly, I was looking into refreshing my C knowledge recently and K&R is still heavily recommended.
K&R is an interesting historical artifact about the basic design decisions of the C language and definitely recommended reading material, but you're not doing yourself a favour using it as reference or for learning the language. For that it is vastly outdated, C ist a much more enjoyable and powerful language since C99.
Re: Writing a C compiler in 500 lines of Python (2023)
#55After many years of programming in other languages, I finally learned C, and came to realize that there aren't actually any compilers that implement all of the C spec. Even GCC and Clang have their grey areas and their bugs. Before this, I had thought that C was a simple language. An idea propped up by articles likes this, as well as the oft touted fact that nearly every embedded system has a C compiler; no matter wh…
That said, "C" (C99 with benign bits of c11 required for modern hardware programming) is only the "less worse" compromise for a computer language: its syntax is already way too rich (I let you think about the beyond sanity computer language syntax out there, yep... even for currently hyped ones).
For instance: C should have only one loop keyword loop{}, finally a real hard compile time constant definition, no integer promotion, no implicit cast (unless void* or some generic number literals), probably explicit static/dynamic casts (but certainly not with the c++ syntax), sized types should be primitives and not the other way around (s32,u32,f64...), no typedef/typeof/generic/etc, but should include inline support for modern hardware architecture programming aka memory barriers/"atomics"/explicit memory unaligned access, etc.
The benchmark is a small team of average devs should be able to develop a real-life compiler in reasonable amount of time.
Ofc, the more the merrier.
Re: Writing a C compiler in 500 lines of Python (2023)
#56After many years of programming in other languages, I finally learned C, and came to realize that there aren't actually any compilers that implement all of the C spec. Even GCC and Clang have their grey areas and their bugs. Before this, I had thought that C was a simple language. An idea propped up by articles likes this, as well as the oft touted fact that nearly every embedded system has a C compiler; no matter wh…
I think the main reason for this is that the C spec was always just an attempt to somewhat harmonize the already existing features of different C compilers, e.g. implementations come first and then after one or two decades, the C committee tries to standardize the features that have survived. That doesn't mean that all C compiler vendors immediatedly hop on board.
But then there's of course MSVC which after a few hopeful years of modernizing their C frontend now seems to have abandondend it again (I wonder though if MSVC as a whole has been abandondend looking at the abundance of red for MSVC in the C++23 and C++26 tables: https://en.cppreference.com/w/cpp/compiler_support.html)
While looking like a weird approach at first, it has definitely worked better than the C++ committee's approach to accept 'ideas' into the standard without even a proof-of-concept implementation - e.g. the one good thing about C++ is that it acts as a filter to stupid ideas before they can make it into C ;)
Re: Writing a C compiler in 500 lines of Python (2023)
#57Earlier quoted context omitted.
I could probably do it - but you wouldn't like it. My dictionaries would be a linked-list, looking for a key becomes a linear search... (if you gave me C++ I'd use std::map) I'm assuming you will allow me to use the C standard library, if I have to implement strlen or malloc in that 500 lines of C I'm not sure I can pull that off. 500 lines is aggressive, but IOCCC gives me plenty of tricks to get the line count down…
A hash table in C is about 30 lines of code, so I don't think you have to stick to linked lists for dictionaries.
// Hashtable definition
#include
#define N 1024
int* map_ptr(const char **keys, int *values, const char *key){
size_t h = 0, c = 0, i;
for (const char *c = key; *c; c++) h = h * 33 + *(unsigned char*)c;
for (i = h % N; c
int main(){
// Set some values
*map_ptr(keys, values, "one") = 1;
*map_ptr(keys, values, "two") = 2;
*map_ptr(keys, values, "three") = 3;
// Retrieve values
printf("one: %i\n", *map_ptr(keys, values, "one"));
printf("two: %i\n", *map_ptr(keys, values, "two"));
printf("three: %i\n", *map_ptr(keys, values, "three"));
return 0;
}Re: Writing a C compiler in 500 lines of Python (2023)
#58Earlier quoted context omitted.
A language is not inherently interpreted or compiled. Some languages are more or less easy to compile efficiently and without embedding a JIT compiler, but any language can be compiled. For Python in particular, there are already compilers. If you want a nightmarish language to compile, look at Ruby. There are compilers even for Ruby.
Python has the same amount of nightmare. Maybe even more. You can add static class and instance accessors at runtime, it supports full monkey patching just like Ruby does. You can meta program modules, classes, objects, you can decorate classes and functions, declare functions and lambdas anywhere. "compilers" usually disallow monkey business and compile only a subset.
Re: Writing a C compiler in 500 lines of Python (2023)
#59Earlier quoted context omitted.
That is a myth often spread by folks that think K&R C book is everything there is to know, never opened the ISO C draft PDF, learned the differences between POSIX and standard library, tried to make their code portable outside GCC or nowadays clang, or even checked the extensions chapter on the compiler.
Who told that? I still hear K&R being recommended as introduction material. If you want to write portable/production-grade C code, well definitely need to study another references.
This is a more useful book for modern days,
https://www.manning.com/books/modern-c
flohofwoe already put it out clearly on his comment.
Re: Writing a C compiler in 500 lines of Python (2023)
#60Earlier quoted context omitted.
Who told that? I still hear K&R being recommended as introduction material. If you want to write portable/production-grade C code, well definitely need to study another references.
Exactly, I was looking into refreshing my C knowledge recently and K&R is still heavily recommended.