A compiler spends almost all of its life telling people what is wrong with their program. Most programs a person writes do not compile on the first try. If the only thing your compiler can say is a byte offset, you have written a tool that is correct and unpleasant to use, and unpleasant tools get abandoned before they get better.
Look at the two blocks above again. Both of them are true. Both of them come from the same failed compile of the same file. The first one is what you get for free the moment you know where in the source string something broke: a single integer. The second one is a file name, a one based line and column, the actual text of that line, and a caret sitting directly under the character that should have come next. Getting from the first to the second is the entire chapter.
One more thing before the code. Every stage of izvor, the lexer, the parser, and later the type checker and the evaluator, is going to need to report an error at some point. If each stage invents its own way of printing one, you end up with four slightly different error formats and four places to fix when you want to change one detail. Instead there is exactly one function that knows how to print an error, and every stage calls it.
Checkpoint
This already works, because diag.c already exists in the repo. The rest of this chapter is you understanding, and being able to rebuild, what just ran.
diag.h is the whole public interface. Three functions is all any stage of the compiler ever needs to know about.
src/diag.h · the entire public interface
void diag_set_path(const char *path);Names the file that errors are reported against. The driver calls this once before compiling, and it defaults to "<input>" when nothing sets it, which is why you never see a crash from a missing file name, just a slightly unhelpful one.
void diag_line_col(const char *src, long offset, int *line, int *col);Converts a byte offset into a one based line and column. Offsets past the end of the source clamp to the end, because that is where TOK_EOF lives, sitting on the zero byte that ends every C string.
void diag_error(const char *src, long offset, const char *fmt, ...)The one function every stage calls. fmt and the ... after it mean this works exactly like printf: pass a format string and however many arguments it needs.
__attribute__((format(printf, 3, 4)));This line is not decoration. It tells clang that argument 3 is a printf style format string and argument 4 onward are its values, so if you write diag_error(src, off, "%d", "oops") the compiler catches the mismatched type at build time instead of you finding it in the terminal output.
Notice what is missing. There is no function to decide whether compilation should stop
after an error. diag_error writes to stderr and returns. Whatever called it
decides what happens next. Printing a message and deciding whether to keep going are two
different jobs, and giving them to two different pieces of code means you can change one
without touching the other.
Checkpoint
Before any of the actual math, diag.c needs to know what file it is talking about. It keeps that in a single file scope variable rather than asking every caller to supply it.
src/diag.c · top of the file, and diag_set_path itself
// The file name printed in the --> line. The parser never learns whatThis comment is the reason for the whole design. Read it with the next two lines.
// file it is reading, so the driver leaves the name here instead of// threading a path through every function that might report an error.static const char *current_path = "<input>";static here means file scope: nothing outside diag.c can see this variable or change it directly. It starts with a sensible default so a program that forgets to call diag_set_path still prints something rather than a null pointer.
void diag_set_path(const char *path) { current_path = (path == NULL) ? "<input>" : path;A ternary, read as one sentence: if path is NULL, use the default, otherwise use path. This is the only place NULL is handled, so nowhere else in diag.c has to worry about it.
}Here is the actual reason for all of this. The parser only ever sees a string of source text. It has no idea whether that text came from a file called main.iz, from a command line flag, or from a test harness that never touched disk. Making the parser carry a file name around, just so it can hand it back when something goes wrong, would mean threading one extra argument through every function between main and the error site. Instead the one place that does know the file name, main, tells diag.c once, and every later error reads it from there.
Checkpoint
Before diag.c can turn an offset into a line and column, it has to survive an offset that does not point at a real character at all.
src/diag.c · clamp, then diag_line_col
// Walk to the NUL to find the length, then pull the offset inside it.// An offset one past the end is normal: that is where TOK_EOF lives.This is the whole trap in one comment. A token stream always ends with an EOF token, and that token's offset points at the zero byte that terminates the source string, one past the last real character. clamp exists so that is not treated as a bug.
static long clamp(const char *src, long offset) { long length = 0; while (src[length] != '\0') { length++; } if (offset < 0) {A negative offset should never happen, but this function does not trust that. If it ever does, land on the start of the file rather than reading memory before the buffer.
return 0; } if (offset > length) {Anything past the end, not just exactly one past it, clamps to the end. This is what stops a corrupted or wrong offset from turning into an out of bounds read.
return length; } return offset;}Now the actual conversion. Given a clamped offset, count line breaks from the start of the file up to that point.
// Counting from the start on every error is O(n) per message. Errors// are rare and the source fits in memory, so a line table would beO(n) means the cost grows in proportion to how far into the file the offset is. A real compiler reporting thousands of errors against a huge file might build a table of where every line starts once, so each lookup is instant. izvor reports a handful of errors against files that fit in memory, so that table would be code you wrote and tested for a cost you never paid.
// machinery with nothing to pay for it.void diag_line_col(const char *src, long offset, int *line, int *col) { offset = clamp(src, offset); int l = 1; int c = 1;Both start at one, not zero, because that is how people count lines and columns. Column one is the first character, not the character before it.
for (long i = 0; i < offset; i++) { if (src[i] == '\n') {Every newline byte crossed on the way to the offset means one more line and a column reset. This is the entire algorithm: walk forward, count newlines.
l++; c = 1; } else { c++;test_diag_1.c pins this against the cases that are easy to get wrong by one: the last character of a line, the newline itself, an empty line in the middle of the file, a file that is nothing but newlines, and the empty file. Every one of those is a real assertion in that file right now, not a hypothetical.
Checkpoint
diag_line_col tells you which line and column an offset falls on. line_bounds answers a different question: given that offset, where does its line start, and how long is it.
src/diag.c · line_bounds
static void line_bounds(const char *src, long offset, long *start, long *length) {Finds where the line holding offset starts, and how long it is, not counting the newline that ends it. Two out parameters rather than a struct, since a struct would be built once per call and thrown away.
long s = offset; while (s > 0 && src[s - 1] != '\n') {Walk backward from the offset until either you hit the start of the file or the character just before you is a newline. Wherever you stop is where this line began.
s--; } long e = offset; while (src[e] != '\0' && src[e] != '\n') {Now walk forward from the same offset until you hit the end of the file or a newline. That is the same loop, run in the other direction, stopped by the two things that can end a line: the file running out, or an actual line break.
e++; } *start = s;The newline itself is never included in either direction, which is exactly right: you want the text of the line, not the character that ends it.
*length = e - s;}This function does not know or care where in the line the offset actually sits. It only finds the boundaries. diag_error is the one that uses those boundaries to print the line and separately works out where the caret goes.
Checkpoint
Here is a case the first five steps do not cover on their own. What does the caret point at when the error is at the true end of the file, after everything, including the last newline?
src/diag.c · at_end_of_source
Without any correction, an error at the end of the file lands on the zero byte after the final newline. line_bounds would then hand back an empty line, because there is nothing between that newline and the end of the file. A caret under a blank line does not tell anyone anything. This function backs up over any trailing newlines so the caret points at the last line that actually has something on it.
static long at_end_of_source(const char *src, long offset) { if (src[offset] != '\0') {If the offset is not sitting on the terminating zero byte, it is not the end of the file, so there is nothing to fix. Leave it exactly where it is.
return offset; } while (offset > 0 && (src[offset - 1] == '\n' || src[offset - 1] == '\r')) {Both line ending characters are checked, because a file saved on Windows can end in carriage return and newline together. Step back over as many trailing line endings as there are.
offset--; } return offset;}This is presentation, not truth. The real offset really is one past the last newline, and diag_line_col will still report it honestly if you call it directly. at_end_of_source only runs inside diag_error, right before printing, which is why diag_line_col itself does not do this correction. One function answers what is technically true, the other decides what looks right on screen, and they are allowed to disagree.
Checkpoint
Everything so far was a helper. This is the function that actually prints, and it is the longest one in the file, so take it in four pieces.
src/diag.c · diag_error
The message itself, using the same offset pipeline the last two steps built.
void diag_error(const char *src, long offset, const char *fmt, ...) {The three dots mean this function takes a variable number of extra arguments, exactly like printf does. There is no fixed limit on how many.
offset = at_end_of_source(src, clamp(src, offset));Both of the last two steps' functions, called on one line. Clamp first so at_end_of_source never reads outside the buffer, then correct for a trailing newline.
int line; int col; diag_line_col(src, offset, &line, &col); va_list args;va_list is C's type for a handle onto a function's variable arguments. va_start points it at the arguments that come after fmt, and va_end releases it. You always pair the two.
va_start(args, fmt); fprintf(stderr, "error: "); vfprintf(stderr, fmt, args);vfprintf is the version of fprintf that takes a va_list instead of the arguments directly. It exists because you cannot forward ... straight through from one function to another, you have to capture it in a va_list first and hand that along instead.
fputc('\n', stderr); va_end(args);Next, the gutter. Every following line has a left margin exactly as wide as the line number, so the vertical bars line up whether the error is on line 3 or line 3000.
char number[16]; int width = snprintf(number, sizeof number, "%d", line);snprintf does two jobs on one line here. It writes the line number into number as text, and it returns how many characters that took, which becomes the gutter width. One call, two answers, no separate step to measure the string afterward.
Then the two header lines, using a printf trick worth knowing on its own.
fprintf(stderr, "%*s--> %s:%d:%d\n", width, "", current_path, line, col);%*s means: print a string, and take its field width from the next argument rather than from the format string itself. Passing width and then an empty string prints exactly width spaces. That is how the arrow line indents to match the gutter before current_path even shows up.
fprintf(stderr, "%*s|\n", width + 1, "");Then the source line itself, byte by byte, with the one substitution that keeps the caret honest.
long start; long length; line_bounds(src, offset, &start, &length); fprintf(stderr, "%s | ", number); for (long i = 0; i < length; i++) { // A tab printed as a tab would shove the caret out of line, so itHere is the tab to space trick promised at the top of this step.
// goes out as one space and one byte stays worth one column. char c = src[start + i];A terminal renders a tab as anywhere from two to eight columns wide depending on its own settings, which the caret line below has no way to match. Printing every tab as a single space keeps one source byte worth exactly one printed column, so the count the caret needs stays simple.
fputc(c == '\t' ? ' ' : c, stderr); } fputc('\n', stderr);
tests/golden/caret-after-tab.iz starts with a real tab character before 12 * (3
+. Its pinned output is this, exactly:
Look at the line starting 1 |. There are two spaces after the bar: one is
the ordinary space the format string always prints, the other used to be a tab. If that
substitution were missing, your terminal would render the real tab as several columns
wide and the caret below would point at the wrong character, while the code would look
completely correct.
Finally, the caret line.
fprintf(stderr, "%*s| %*s^\n", width + 1, "", col - 1, "");Same %*s trick, twice. First pad to the gutter width so the bar lines up, then pad col - 1 spaces before the caret. Column is one based, so a column 1 error needs zero spaces before the caret and column 8 needs seven, which is exactly what subtracting one gives you.
Checkpoint
diag.c does not know what a parser is. It only knows how to turn an offset and a message into printed text. The parser is what supplies the offset and the words.
src/parser.c · error_at
static void error_at(const Parser *p, const char *msg) {diag owns the formatting. This function's only job is supplying the two facts diag needs: where, and what should have been there instead.
long offset = p->current.start - p->lexer.src;The token the parser is currently sitting on carries a pointer, start, directly into the source string. Subtracting the base of that same string, p->lexer.src, turns a pointer into the plain integer offset that every diag function expects.
diag_error(p->lexer.src, offset, "%s, found %s",The message format is fixed here, once. Every call site below supplies only what it expected, and token_type_name supplies what it actually found.
msg, token_type_name(p->current.type));}Three places call it, and each one is a golden test you already have.
error_at(p, "expected ')'");This is what fires for tests/golden/unclosed-paren.iz and tests/golden/deep-in-the-file.iz: an open paren that never sees its close.
error_at(p, "expected expression");This is what fires for tests/golden/empty.iz, tests/golden/missing-operand.iz, and tests/golden/caret-after-tab.iz: the parser wanted a number, a paren, or a minus sign, and the token in front of it was none of those.
error_at(p, "expected end of input");This is what fires for tests/golden/trailing-tokens.iz and tests/golden/unknown-character.iz: the expression parsed fine, and then the file kept going.
And main.c is the one place that actually knows a file name, which is why it is the only caller of diag_set_path.
diag_set_path(argv[1]);Compiling a real file, so the path is whatever the user typed on the command line.
diag_set_path("<command line>");izvor also accepts -e "<expression>" for a one line program with no file behind it, so this branch names it honestly rather than printing a made up path.
Checkpoint
A unit test usually checks a return value. An error message is not a return value, it is the whole product for the person reading it, so it gets a different kind of test.
tests/golden · pairs of .iz and .expected files
Every file in tests/golden comes in a pair. Here is the smallest one, an empty program, shown next to the exact bytes izvor is expected to print for it.
Even an empty file gets a real line and column, because clamp treats offset zero into an empty string as valid, and diag_line_col starts counting at line one, column one, before it has looked at a single byte.
# Golden tests pin the exact text of compiler errors. A diagnostic is aThis comment is the reason the target exists, and it is worth reading twice.
# user interface, so changing one should be a deliberate act that shows# up in a diff, not a side effect noticed by nobody.golden: $(BUILD)/izvorDepends on the built compiler, so this target always runs against the code you just changed.
@failed=0; \ for f in $(GOLDEN); do \ expected="$${f%.iz}.expected"; \ actual="$(BUILD)/$$(basename $${f%.iz}).actual"; \ ./$(BUILD)/izvor "$$f" > "$$actual" 2>&1 || true; \The || true matters. izvor exits non zero on a parse error, and without this, make would treat that as the recipe failing and stop before the diff ever ran.
if ! diff -u "$$expected" "$$actual"; then \diff -u prints exactly which lines differ, so a broken golden test tells you the wrong caret position or the wrong word, not just that something failed.
echo "golden: $$f does not match $$expected"; failed=1; \ fi; \ done; \ if [ $$failed -ne 0 ]; then exit 1; fi; \Every pair runs even after one fails, and the target only fails at the very end, so one broken message never hides another one right next to it.
echo "golden: $(words $(GOLDEN)) cases match"The nine pairs currently in the repo were each written to catch one specific way an error message can drift: an unclosed paren, a missing operand, a file that ends mid expression, a tab before the failure, an error buried on line three of three, an empty file, trailing tokens after a valid expression, and an unknown character. Add a tenth case the same way: an .iz file with the input, and an .expected file with the exact stderr you want, byte for byte.
Checkpoint
Golden tests check that izvor reports the errors you thought to write down. The fuzzer checks the much narrower, much more important claim that it never crashes on input you never thought of at all.
tests/fuzz.c
/* Fuzz harness for the front end.Read the whole comment. It says exactly what is being tested and exactly what is not, and both halves matter.
The claim being tested is narrow and worth stating exactly: for ANY input string, lexing and parsing must either return a tree or return NULL, and must never read out of bounds, never overflow, and neverNotice what the claim does not say. It does not say the tree is correct. It says the compiler does not corrupt memory while trying, on any input at all, including ones no test author would think to write.
crash. UndefinedBehaviorSanitizer is what turns those "never"s into a non-zero exit, so this binary is only meaningful when built with it. The generator is a fixed-seed xorshift rather than rand(), so a failureA fixed seed means the exact same sequence of random inputs runs every single time, on every machine. If CI finds a crash on input number 4821, you can reproduce input number 4821 on your own laptop, instead of chasing a bug that only shows up sometimes.
on CI reproduces byte for byte on a laptop. A failing run prints the input that broke it before it dies. Deliberately not fuzzed: eval(). Evaluation of arbitrary trees canThis is the honest part. eval is skipped on purpose, not by oversight.
overflow a long, which is real undefined behavior and a real gap, but it is a semantics gap rather than a parsing one. It is tracked in docs/ROADMAP.md under checked arithmetic. For the same reason the generator caps runs of digits, since the lexer folds digits into aA known, tracked gap, kept out of a test that would otherwise report it as a crash every single run. Fuzzing eval would just rediscover the same known bug twenty thousand times a build, which is noise, not information.
long as it scans and a 40-digit literal would overflow before the parser ever saw it. */The random number generator is deliberately small and deliberately not the standard library's.
static unsigned int state = 0x9E3779B9u; static unsigned int next_random(void) {This is a xorshift generator: three shifts and three xors against its own previous output. It is not cryptographically random, and it does not need to be. It only needs to produce a long, varied sequence of bytes deterministically.
state ^= state << 13; state ^= state >> 17; state ^= state << 5; return state;}20000 strings, each built from an alphabet that mixes real izvor syntax with junk the lexer has to reject, get lexed and parsed, and every resulting tree gets freed immediately so LeakSanitizer, running separately under make asan, can catch a half built tree with a missing free.
Checkpoint
Every push runs the whole suite on two operating systems, plus a second job that only runs on one of them, for a reason worth being honest about.
.github/workflows/ci.yml
# Two platforms because the compiler is meant to be portable C11 andLinux and macOS use different C libraries and different versions of clang, and code that only ever compiled on one of them has a way of turning out to be less portable than it looked.
# the two toolchains disagree about enough to be worth knowing. strategy: fail-fast: falseWithout this, GitHub Actions cancels the macOS job the moment the Linux job fails, or the other way around. fail-fast: false lets both finish, so one push tells you about problems on both platforms instead of just the first one to fail.
matrix: os: [ubuntu-latest, macos-latest] - name: BuildEXTRA_CFLAGS=-Werror here turns every warning into a build failure. On a laptop, that flag is off, so work in progress still compiles while you are in the middle of it. CI is where sloppy code actually gets stopped.
run: make EXTRA_CFLAGS=-Werror all - name: Unit and golden tests run: make EXTRA_CFLAGS=-Werror test - name: Fuzz the front end run: make EXTRA_CFLAGS=-Werror fuzz # AddressSanitizer catches leaks and use-after-free, and its runtimeThis is the honest part of the chapter.
# deadlocks on startup under Apple clang, so this half of the checking # only runs on Linux. sanitizers: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - name: Address and LeakSanitizerOnly this one job builds with AddressSanitizer, and only on Linux.
run: make asanI tried the macOS leaks tool for this before settling on Linux only AddressSanitizer. I removed it. I deliberately leaked a block of memory to check that the tool actually caught anything, and it reported zero leaks. The process was not debuggable under the current macOS security policy, so leaks had nothing to inspect and just said everything was fine. A check that reports success no matter what is worse than no check at all, because it looks like coverage on a dashboard while catching nothing. That is why the Makefile only turns AddressSanitizer on where it actually works, and why CI only runs that job on ubuntu-latest.
The asan target rebuilds everything from clean with AddressSanitizer and LeakSanitizer turned on, then runs the same test and fuzz targets you already ran above.
# The front end allocates one Node per tree node and nothing else, soThis is why LeakSanitizer is worth running at all: the shape of what could go wrong is narrow and known.
# anything LeakSanitizer reports is a missing ast_free. Recursing into# make keeps this a one-word command rather than a flag to remember.asan: $(MAKE) clean $(MAKE) EXTRA_CFLAGS="-fsanitize=address -fno-omit-frame-pointer" test fuzzA fresh recursive make, because the object files built for the normal, faster undefined behavior only build are not safe to reuse with a completely different sanitizer linked in.
$(MAKE) cleanCheckpoint
Slower than plain make test, because it rebuilds everything from clean twice. Run it before you push, not on every save.