Statements and Variables

No code. This is the idea the rest of the chapter builds, and it is worth five minutes.

Everything izvor has handled so far is an expression, which means a piece of syntax that produces a value. 2 + 3 produces five. (4 - 1) produces three. Expressions nest inside each other, which is exactly why your parser from chapter two is recursive.

let x = 10; produces nothing. It is a statement, which means a piece of syntax that happens. Its whole point is the side effect, which is that afterwards the name x means ten. Statements sit in a flat sequence instead of nesting, so a program becomes a list rather than a single tree.

That split is why a variable is harder than it looks. In print x + y; the name x is a leaf sitting exactly where a number sits. But the parser cannot know what it means. Only at run time, after the earlier statements have happened, does anything know that x is ten. So you need a second structure alongside the tree, a place where names map to values. That structure is called an environment, and building it is step 9.

The decision this chapter depends on

Statements end with a semicolon. Required, not optional. I picked it at the end of chapter four, and the reason is worth repeating: without a terminator the parser cannot tell where one statement ends once a bare expression becomes a legal statement, and adding a terminator to a language later breaks every program already written in it.

Checkpoint

Start from a clean tree so you can see exactly what this chapter changes.

$ cd ~/Developer/izvor $ git status $ make EXTRA_CFLAGS=-Werror test

Everything should pass before you add anything.

Clean tree, all tests passing.

A program is a sequence of statements and you do not know how many until you have parsed them all. C has no growable array, so you write one. It is about twenty lines and you will reuse it for call arguments and function parameters in chapter seven.

src/ast.h · directly above the line Node *ast_number(long value);

src/ast.hadd 12 lines
1typedef struct {Put this after the struct Node definition, because it holds pointers to Nodes.
2    Node **items;Two stars. It is a pointer to an array of Node pointers. The array itself lives on the heap so it can be reallocated when it fills up.
3    int count;How many are in it.
4    int capacity;How many it can hold before it has to grow. Keeping these two separate is what lets you avoid calling realloc on every single push.
5} NodeList;
6 
7void nodelist_init(NodeList *list);
8int  nodelist_push(NodeList *list, Node *node);Returns 0 if the allocation failed, 1 if it worked. Every caller has to check it, the same way the constructors already check malloc.
9void nodelist_free(NodeList *list);Frees the nodes inside it as well as the array. Write that down in a comment, because a reader cannot tell from the name whether the list owns what it holds, and getting that wrong is a double free or a leak.

src/ast.c · at the bottom of the file, after ast_free

src/ast.cadd 26 lines
1void nodelist_init(NodeList *list) {
  list->items = NULL;Starting at NULL with zero capacity is deliberate. realloc(NULL, n) is defined to behave exactly like malloc(n), so the growth code below needs no special case for the first push.
  list->count = 0;
  list->capacity = 0;
5}

7int nodelist_push(NodeList *list, Node *node) {
  if (list->count == list->capacity) {Only grow when it is actually full.
      int capacity = list->capacity < 8 ? 8 : list->capacity * 2;Doubling, with a floor of 8. Doubling is what keeps this cheap: pushing n items costs about 2n copies in total rather than n squared, because each item moves only a handful of times no matter how big the list gets.
      Node **items = realloc(list->items, (size_t)capacity * sizeof(Node *));Into a new variable, never straight back into list->items. If realloc fails it returns NULL and leaves the old block alive, so assigning directly would lose the only pointer to everything you already had.
      if (items == NULL) {
          return 0;
      }
      list->items = items;
      list->capacity = capacity;
  }
  list->items[list->count++] = node;
  return 1;
19}

21void nodelist_free(NodeList *list) {
  for (int i = 0; i < list->count; i++) {The list owns its nodes, so it frees them.
      ast_free(list->items[i]);
  }
  free(list->items);
  nodelist_init(list);Reset it to empty rather than leaving items pointing at freed memory. Using the freed pointer afterwards is one of the nastiest bugs in C, because it usually appears to work.
27}

Checkpoint

$ make EXTRA_CFLAGS=-Werror test

You should get a clean build and all existing tests passing. Nothing calls the list yet.

NodeList compiles.

One of these additions is not about statements at all. It is the thing that lets every later chapter produce a good error message.

src/ast.h · in the typedef enum, after NODE_BINARY

src/ast.hadd 6 lines
  NODE_BINARY,
  NODE_IDENT,A name used in an expression, like the x in print x + y;. This is an expression, not a statement.
  NODE_LET,A binding that cannot be changed afterwards.
  NODE_VAR,A binding that can.
  NODE_ASSIGN,Changing an existing binding, x = 5;. Separate from NODE_VAR because declaring a name and reassigning one are different events, and the checks on them differ.
  NODE_PRINT,
  NODE_EXPR_STMTAn expression on its own followed by a semicolon. Useless today, since nothing has a side effect yet, but function calls in chapter seven need it and it costs nothing to put in now.

src/ast.h · inside struct Node, immediately after NodeType type;

src/ast.hadd 1 line, then 2 union members

1

    long offset;

Where in the source this node came from, as a byte offset. This is the whole point of the step. Your diagnostics module already turns an offset into a file, a line, a column and a caret, so once every node carries one, every later pass can report a mistake against the exact characters that caused it. Without this, a type error in chapter six can only say "somewhere in your program".

src/ast.hinside the union, after the binary member

1

        struct { const char *name; int length; } ident;

A name borrows its text from the source, exactly as a Token does. Nothing is copied and nothing is freed, and the same rule applies: the source has to outlive the tree.

2

        struct { const char *name; int length; Node *value; } binding;

Shared by NODE_LET, NODE_VAR and NODE_ASSIGN. All three are a name and the expression being put into it, so they get one union member rather than three identical ones.

3

        struct { Node *value; } stmt;

Shared by NODE_PRINT and NODE_EXPR_STMT, which are both just a wrapper around one expression.

src/ast.h · with the other constructor declarations

src/ast.hadd 3 lines

1

Node *ast_ident(const char *name, int length);

2

Node *ast_binding(NodeType type, const char *name, int length, Node *value);

Taking the kind as a parameter, so one function covers let, var and assign. Do not write three near-identical constructors.

3

Node *ast_stmt(NodeType type, Node *value);

Checkpoint

$ make EXTRA_CFLAGS=-Werror test

This should fail. You added six values to NodeType and the switch in ast_free has no default:, so clang lists every one you have not handled. That is step 5's work, and the compiler finding it for you is exactly why there is no default case.

Build fails, naming the six unhandled node kinds.

Same shape as the three you already have. Copy their failure checks exactly.

src/ast.c · after ast_binary, before ast_free

src/ast.cadd 23 lines
1Node *ast_ident(const char *name, int length) {
  Node *node = new_node(NODE_IDENT);
  if (node == NULL) {Same failure check as every other constructor in this file. Do not skip it because the node is small.
      return NULL;
  }
  node->as.ident.name = name;
  node->as.ident.length = length;
  return node;
9}

11Node *ast_binding(NodeType type, const char *name, int length, Node *value) {The caller passes NODE_LET, NODE_VAR or NODE_ASSIGN. Nothing here checks which, because the parser is the only caller and it already knows.
  Node *node = new_node(type);
  if (node == NULL) {
      return NULL;
  }
  node->as.binding.name = name;
  node->as.binding.length = length;
  node->as.binding.value = value;
  return node;
20}

22Node *ast_stmt(NodeType type, Node *value) {
  Node *node = new_node(type);
  if (node == NULL) {
      return NULL;
  }
  node->as.stmt.value = value;
  return node;
29}

src/ast.c · inside new_node, after node->type = type;

src/ast.cadd 1 line

1

    node->offset = 0;

Every node starts with no position and the parser fills it in, in step 6. Zeroing it here means a node that nobody sets a position on reports the start of the file rather than reading uninitialised memory, which the sanitizer would catch but which is better not to write in the first place.

Checkpoint

$ make EXTRA_CFLAGS=-Werror test

Still failing on the same unhandled cases in ast_free. That is next.

Constructors written, same failure as before.

Six cases, and getting one wrong is either a leak or a crash. The compiler told you exactly which six.

src/ast.c · in the switch inside ast_free, after the NODE_BINARY case

src/ast.cadd 11 lines
  case NODE_IDENT:A name has no children, and its text is borrowed from the source rather than owned. So there is nothing to free, exactly like NODE_NUMBER.
      break;
  case NODE_LET:Three labels falling into one body. This is legal C and it is the right tool here, because all three use the same union member and so free the same child.
  case NODE_VAR:
  case NODE_ASSIGN:
      ast_free(node->as.binding.value);
      break;
  case NODE_PRINT:
  case NODE_EXPR_STMT:
      ast_free(node->as.stmt.value);
      break;

Trap 1 of 3 · the union is why you check the tag first

Every node is the same size and the union members sit on top of each other in the same bytes. So node->as.binding.value on a NODE_NUMBER would read the number as if it were a pointer, and then free it. That is not a crash you can debug easily, it is memory corruption that shows up somewhere else later.

The tag is the only thing that says which member is the real one. Check it before you touch the union, every time, with no exceptions. This is the cost of a tagged union and it is why the compiler listing your unhandled cases is worth protecting.

Checkpoint

$ make EXTRA_CFLAGS=-Werror test

Clean build now, all existing tests passing. The tree can hold statements even though nothing produces them yet.

Builds clean again.

One tiny helper, then one edit per ast_ call in the parser. Boring, and it is what makes every error message in chapters six through twelve possible.

src/parser.c · directly under error_at

src/parser.cadd 8 lines
1/* at - stamp a node with the source offset it came from, and pass it
2   straight through, so a construction stays one expression. Tolerates
3   NULL so allocation failures keep flowing to the caller. */
4static Node *at(Node *node, long offset) {Wrapping instead of adding an offset parameter to all six constructors. That keeps ast.c unchanged and, more importantly, keeps the chapter two listing accurate. Changing a published interface to avoid one wrapper is a bad trade.
5    if (node != NULL) {The NULL check is what lets you write return at(ast_number(v), start); without thinking. If the allocation failed you still return NULL and the caller's existing error path handles it.
6        node->offset = offset;
7    }
8    return node;
9}

Now wrap every construction in the parser. The offset you want is the position of the token that started that piece of syntax, which is p->current.start - p->lexer.src taken before you consume it. In parse_factor the number case becomes this:

src/parser.cparse_factor, the TOK_NUMBER branch

1

        long offset = p->current.start - p->lexer.src;

Capture it first. After parser_advance the current token is the next one and the position is gone.

2

        long value = p->current.value;

3

        parser_advance(p);

4

        return at(ast_number(value), offset);

Do the same for the unary minus in parse_factor, and for the ast_binary calls in parse_term and parse_expression. For a binary node, capture the offset of the operator token before you advance past it, because "the error is at this +" is what a person wants to be shown.

Checkpoint

$ make EXTRA_CFLAGS=-Werror test

All tests still pass, because nothing reads offset yet. You are paying in advance here, and chapter six is where it pays back.

Every ast_ call in parser.c is wrapped in at().

Four lines. A name goes exactly where a number goes, because both are leaves.

src/parser.c · in parse_factor, directly after the TOK_NUMBER block and before the TOK_LPAREN one

src/parser.cadd 7 lines
  if (parser_check(p, TOK_IDENT)) {
      long offset = p->current.start - p->lexer.src;
      const char *name = p->current.start;Copying the pointer and the length out of the token, not the characters. The tree borrows from the source exactly as the token does.
      int length = p->current.length;
      parser_advance(p);
      return at(ast_ident(name, length), offset);
  }

Putting it in parse_factor rather than anywhere else is the whole design. parse_factor is the bottom of the precedence chain, the place that handles things that cannot be broken down further. A name is one of those. Because it lives there, x * 2 + 1 gets the right shape for free, with no new rules about how names combine with operators.

Checkpoint

$ make EXTRA_CFLAGS=-Werror test $ make && ./build/izvor -e "x + 1"

The parse should now succeed and print a tree. Evaluating it will not work yet, because eval in main.c has no case for NODE_IDENT, and the compiler will have told you so during the build. Leave it broken until step 10.

A name parses as an expression.

The biggest step in the chapter, and it contains one genuinely clever piece that every real parser uses.

Here is the grammar you are implementing. Write it into a comment above the code.

the grammarfor the comment
1program   -> statement* EOF
2statement -> "let" IDENT "=" expression ";"
3           | "var" IDENT "=" expression ";"
4           | "print" expression ";"
5           | expression ( "=" expression )? ";"That last line is the trick. Read the trap below before you write it.

Trap 2 of 3 · why assignment is not its own rule

The obvious grammar has a rule IDENT "=" expression ";" for assignment and another for a bare expression statement. Both start with a name, so to tell them apart your parser would have to look at the second token, and it only holds one.

You could add a second lookahead token. Do not. The standard fix is smaller: parse an expression normally, and then check whether an = follows. If it does, look at what you already parsed. If it is a NODE_IDENT, rebuild it as an assignment. If it is anything else, that is an error, because 2 + 3 = x; has no meaning.

This costs nothing and it scales. In chapter eleven, when a[0] = 5; and p.x = 5; become legal, you add two node kinds to the same check instead of rewriting the grammar.

src/parser.c · after parse_expression, before parser_parse

src/parser.cparse_statement, about 45 lines
1static Node *parse_statement(Parser *p) {
  long offset = p->current.start - p->lexer.src;The position of the first token of the statement. Every branch below stamps its node with this, so an error about a statement points at where the statement starts.

  if (parser_check(p, TOK_LET) || parser_check(p, TOK_VAR)) {Both branches are identical except for the node kind, so they share one body.
      NodeType kind = parser_check(p, TOK_LET) ? NODE_LET : NODE_VAR;
      parser_advance(p);

      if (!parser_check(p, TOK_IDENT)) {Check before you consume. If it is not a name, the error should point at whatever is actually there.
          error_at(p, "expected a name after let or var");
          return NULL;
      }
      const char *name = p->current.start;
      int length = p->current.length;
      parser_advance(p);

      if (!parser_match(p, TOK_EQUAL)) {parser_match consumes the token if it is there and reports whether it was. This is exactly what it was written for in chapter two.
          error_at(p, "expected '=' after the name");
          return NULL;
      }

      Node *value = parse_expression(p);
      if (value == NULL) return NULL;
      if (!parser_match(p, TOK_SEMICOLON)) {You have not added TOK_SEMICOLON yet. Step 9 does the lexer side. If you want this to compile now, add it to the enum, to token_type_name, and a case ';' to the switch in lexer_next, exactly the way you added = in chapter four.
          error_at(p, "expected ';'");
          ast_free(value);Free what you built before you bail. Every error path in this function has to do this or the parse leaks, and the fuzzer from chapter three will find it.
          return NULL;
      }
      return at(ast_binding(kind, name, length, value), offset);
  }

  if (parser_match(p, TOK_PRINT)) {
      Node *value = parse_expression(p);
      if (value == NULL) return NULL;
      if (!parser_match(p, TOK_SEMICOLON)) {
          error_at(p, "expected ';'");
          ast_free(value);
          return NULL;
      }
      return at(ast_stmt(NODE_PRINT, value), offset);
  }

  Node *expr = parse_expression(p);Here is the trick from the trap. Parse an expression first and decide what it was afterwards.
  if (expr == NULL) return NULL;

  if (parser_match(p, TOK_EQUAL)) {
      if (expr->type != NODE_IDENT) {The check that makes 2 + 3 = x; an error instead of nonsense.
          error_at(p, "cannot assign to this");
          ast_free(expr);
          return NULL;
      }
      Node *value = parse_expression(p);
      if (value == NULL) { ast_free(expr); return NULL; }
      if (!parser_match(p, TOK_SEMICOLON)) {
          error_at(p, "expected ';'");
          ast_free(expr); ast_free(value);
          return NULL;
      }
      Node *node = at(ast_binding(NODE_ASSIGN, expr->as.ident.name,Pulling the name back out of the node you already built, then throwing that node away on the next line. Slightly wasteful and much simpler than the alternative.
                                  expr->as.ident.length, value), offset);
      ast_free(expr);Safe, because NODE_IDENT owns nothing. Its name is borrowed from the source, which is still alive, so the name you just handed to the new node stays valid.
      return node;
  }

  if (!parser_match(p, TOK_SEMICOLON)) {
      error_at(p, "expected ';'");
      ast_free(expr);
      return NULL;
  }
  return at(ast_stmt(NODE_EXPR_STMT, expr), offset);
70}

Now replace parser_parse so a program is a list of statements. Change its declaration in src/parser.h to match.

src/parser.cparser_parse, rewritten
1int parser_parse(Parser *p, NodeList *out) {Returns 1 on success and 0 on failure, and writes the statements into a list the caller owns. It no longer returns a Node, because a program is no longer one tree.
  nodelist_init(out);
  while (!parser_check(p, TOK_EOF)) {Keep going until the input runs out. The EOF token is what makes this loop terminate, which is why chapter one gave the end of input a token of its own.
      Node *stmt = parse_statement(p);
      if (stmt == NULL) {
          nodelist_free(out);One bad statement throws the whole program away. That is a deliberate simplification for now. Chapter eight adds recovery so a broken file reports every problem in it instead of only the first.
          return 0;
      }
      if (!nodelist_push(out, stmt)) {Out of memory. Rare, and still handled, because the alternative is losing a statement silently.
          ast_free(stmt);
          nodelist_free(out);
          return 0;
      }
  }
  return 1;
16}

Checkpoint

$ make EXTRA_CFLAGS=-Werror test

tests/test_parser_2.c will now fail to compile, because it calls parser_parse the old way. That is correct and expected. Step 11 rewrites it. If you want a green build in between, comment the file out.

parse_statement and the new parser_parse are written.

Two jobs. One line in the lexer, then a new file that answers "what does this name mean".

src/lexer.h · in the enum, next to TOK_EQUAL

src/lexer.h and src/lexer.c3 edits

1

    TOK_SEMICOLON,

Add it, then add case TOK_SEMICOLON: return "SEMICOLON"; to token_type_name, then case ';': return make_token(lx, TOK_SEMICOLON, start); to the switch at the bottom of lexer_next. Exactly the three places you touched for = in chapter four.

src/env.h · new file

src/env.hnew file, 24 lines
1#ifndef IZVOR_ENV_H
2#define IZVOR_ENV_H
3 
4#include <stdbool.h>
5 
6#define ENV_MAX 256A fixed ceiling, on purpose. 256 variables is far more than any program this language can currently express, and a fixed array means no allocation and nothing to free. When it stops being enough you will know, because step 10 reports it as a real error instead of corrupting memory.
7 
8typedef struct {
9    const char *name;Borrowed from the source, the same as everywhere else. A binding is only valid while the program text is alive.
10    int         length;
11    long        value;
12    bool        mutable;True for var, false for let. This one field is how immutability by default is actually enforced.
13} Binding;
14 
15typedef struct {
16    Binding items[ENV_MAX];
17    int     count;
18} Env;
19 
20void env_init(Env *env);
21Binding *env_find(Env *env, const char *name, int length);Returns a pointer to the binding so the caller can read it or write to it, or NULL if the name is not defined. Handing back a pointer is what makes assignment a one liner in step 10.
22bool env_define(Env *env, const char *name, int length, long value, bool mutable);
23 
24#endif

src/env.c · new file

src/env.cnew file, 24 lines
1#include <string.h>
2#include "env.h"
3 
4void env_init(Env *env) {
5    env->count = 0;
6}
7 
8Binding *env_find(Env *env, const char *name, int length) {
9    for (int i = env->count - 1; i >= 0; i--) {Backwards, from the most recent binding to the oldest. It makes no difference today, but in chapter seven, when an inner scope can shadow an outer one, searching backwards is what makes the nearest definition win.
10        if (env->items[i].length == length &&Length first, then bytes. Exactly the same trap as word_is in chapter four, for exactly the same reason: these names are slices of the source and have no zero byte at the end.
11            memcmp(env->items[i].name, name, (size_t)length) == 0) {
12            return &env->items[i];
13        }
14    }
15    return NULL;
16}
17 
18bool env_define(Env *env, const char *name, int length, long value, bool mutable) {
19    if (env->count >= ENV_MAX) {
20        return false;
21    }
22    env->items[env->count].name = name;
23    env->items[env->count].length = length;
24    env->items[env->count].value = value;
25    env->items[env->count].mutable = mutable;
26    env->count++;
27    return true;
28}

Trap 3 of 3 · a linear search is the right answer here

Looking through every binding to find one name is slow in the way that textbooks warn about. It does not matter. A program with 20 variables does 20 comparisons, and the comparison is a length check that usually fails immediately.

A hash table would be more code, another allocation to manage, and a worse error story, in exchange for a speedup you could not measure. Build the hash table when you have a program slow enough to need it, and not before. Write that reasoning in a comment so the next person, including you in six months, knows it was a decision and not an oversight.

Makefile · the SRC line

Makefile1 edit

1

SRC := src/lexer.c src/ast.c src/parser.c src/diag.c src/env.c

Add src/env.c to the end. Forget this and the build fails at the link stage with "undefined symbols" for every env function, which looks alarming and means nothing more than this.

Checkpoint

$ make EXTRA_CFLAGS=-Werror test

Still failing on main.c and test_parser_2.c, which is the next two steps.

env.c and env.h exist and are in the Makefile.

The driver changes shape. One function per statement kind, and the errors go through diag so they look like every other izvor error.

src/main.c · add the includes, then replace the bottom half of the file

Add #include "env.h" at the top. Then give eval two more parameters, because it now needs to look names up and to report where a mistake was:

src/main.ceval, the NODE_IDENT case
1static long eval(const Node *node, Env *env, const char *src, int *ok) {The ok flag is how a runtime error gets out without calling exit. The old code exited on division by zero, which works but makes the evaluator impossible to test. Set *ok = 0 and return 0 instead, and check it after every recursive call.

  case NODE_IDENT: {
      Binding *b = env_find(env, node->as.ident.name, node->as.ident.length);
      if (b == NULL) {
          diag_error(src, node->offset, "undefined name '%.*s'",Here is the payoff for step 3 and step 6. node->offset puts the caret under the exact name. And %.*s prints a string with a length supplied as an argument, which is how you print text that has no zero byte at the end.
                     node->as.ident.length, node->as.ident.name);
          *ok = 0;
          return 0;
      }
      return b->value;
  }

Then a function that runs one statement:

src/main.cexec_statement, about 40 lines
1static int exec_statement(const Node *node, Env *env, const char *src) {Returns 1 if it worked and 0 if it reported an error. No exit anywhere in this file from now on.
  int ok = 1;
  switch (node->type) {

  case NODE_LET:
  case NODE_VAR: {
      if (env_find(env, node->as.binding.name, node->as.binding.length) != NULL) {Redefining a name in the same scope is an error. Deciding that now, rather than letting the second one quietly win, is a language design choice and it is the one that catches typos.
          diag_error(src, node->offset, "'%.*s' is already defined",
                     node->as.binding.length, node->as.binding.name);
          return 0;
      }
      long v = eval(node->as.binding.value, env, src, &ok);
      if (!ok) return 0;
      if (!env_define(env, node->as.binding.name, node->as.binding.length,The mutable flag is node->type == NODE_VAR, which is the single line that makes let mean something.
                      v, node->type == NODE_VAR)) {
          diag_error(src, node->offset, "too many variables, the limit is %d", ENV_MAX);
          return 0;
      }
      return 1;
  }

  case NODE_ASSIGN: {
      Binding *b = env_find(env, node->as.binding.name, node->as.binding.length);
      if (b == NULL) {
          diag_error(src, node->offset, "undefined name '%.*s'",
                     node->as.binding.length, node->as.binding.name);
          return 0;
      }
      if (!b->mutable) {The other half of immutability. Write the error message carefully, because this one will be read by people who did not choose the rule. Naming var in it turns a refusal into an instruction.
          diag_error(src, node->offset,
                     "'%.*s' was declared with let, so it cannot be changed. Use var instead",
                     node->as.binding.length, node->as.binding.name);
          return 0;
      }
      long v = eval(node->as.binding.value, env, src, &ok);
      if (!ok) return 0;
      b->value = v;One assignment, because env_find handed back a pointer into the array rather than a copy.
      return 1;
  }

  case NODE_PRINT: {
      long v = eval(node->as.stmt.value, env, src, &ok);
      if (!ok) return 0;
      printf("%ld\n", v);
      return 1;
  }

  case NODE_EXPR_STMT:
      (void)eval(node->as.stmt.value, env, src, &ok);The (void) says "I am throwing this away on purpose", which stops -Wextra from asking about the unused result. Being explicit here is better than the warning going quiet by accident.
      return ok;

  default:The one place a default belongs. This switch covers statement kinds, but NodeType also contains expression kinds that can never appear here, so listing them all would be noise. An expression reaching this function is a bug in the parser, not in the user's program, which is why the message says internal error.
      fprintf(stderr, "internal error: not a statement\n");
      return 0;
  }
56}

Then rewrite main to parse into a NodeList, create one Env, and run the statements in order, stopping at the first one that fails. Free the list at the end. The S-expression printer stays useful, so keep it and put it behind a flag if you like.

Checkpoint

This is the milestone. Put this in a file called hello.iz:

$ cat hello.iz let x = 10; let y = 20; print x + y; $ make && ./build/izvor hello.iz

You should see 30. If you do, izvor is a programming language. Try print z; and check the caret lands under the z, and let a = 1; a = 2; and check it tells you to use var.

hello.iz prints 30.

You broke a test file in step 8 and that was correct. Now make it right, and cover what you just built.

tests/test_parser_2.c · then two new files

Every call in test_parser_2.c looked like t = parse(&p, "2+3*4"); and expected a tree back. A program is now a list, so change the local helper to parse into a NodeList and hand back list.items[0], and add a semicolon to every source string in the file. Nothing about what the tests assert changes, only how they get at the tree.

Then write tests/test_env_1.c, covering the environment on its own:

tests/test_env_1.cwhat to assert

1

define then find

A name you defined comes back with the right value, and one you did not returns NULL.

2

prefix names

Define count, then look up co with length 2. It must not match. This is the length check on line 10 of env.c, and it is the same bug as letter matching let in chapter four.

3

shadowing order

Define the same name twice with different values and check the search finds the second one. It proves the loop runs backwards, which is what chapter seven depends on.

4

the ceiling

Define ENV_MAX names, check they all work, then check the next env_define returns false rather than writing past the end of the array.

Then tests/test_parser_3.c for statement shapes:

tests/test_parser_3.cwhat to assert

1

"let x = 1; let y = 2;"

Two statements in the list, both NODE_LET, in that order.

2

"var x = 1; x = 2;"

NODE_VAR then NODE_ASSIGN. Proves the lookahead trick in step 8 works.

3

"print x + y;"

NODE_PRINT wrapping a NODE_BINARY. Also check the offsets: the print node's offset should be 0 and the identifier's should be 6.

4

the failures

"let x = 1" with no semicolon, "let = 1;" with no name, "2 + 3 = x;" assigning to something that is not a name, and "" which should succeed with an empty list rather than fail. That last one is the case people forget.

Checkpoint

$ make EXTRA_CFLAGS=-Werror test $ make fuzz

Run the fuzzer too. You added error paths that free things, and freeing the same node twice is exactly what it is built to catch. It will need one change: it calls parser_parse the old way, so update it to use a NodeList and call nodelist_free.

All tests pass and the fuzzer is clean.

Four new error messages went in this chapter. Golden tests are how they stay good.

tests/golden/ · four pairs of files

For each one, write the .iz file, run ./build/izvor tests/golden/<name>.iz, read the output carefully, and only once you are happy with the wording save it as the matching .expected file. Reading it first is the point. A golden test freezes whatever you give it, including a badly worded message.

tests/golden/4 new cases

1

undefined-name.iz

print z;. Check the caret is under the z and not under print.

2

assign-to-let.iz

let a = 1;\na = 2;. Check it reports line 2 and names var.

3

missing-semicolon.iz

let x = 1. Check the caret lands at the end of the line rather than on a blank line, which is the end-of-file handling from chapter three doing its job.

4

bad-assign-target.iz

2 + 3 = x;.

Last thing, and do not skip it. Open docs/ROADMAP.md and move phase 3 to done, with a sentence on what actually landed. Then open docs/ARCHITECTURE.md and add the three decisions you made in this chapter: semicolons are required, redefining a name in the same scope is an error, and the environment is a flat array with a fixed ceiling of 256 because a hash table was not worth it yet.

That last one matters more than it looks. In an interview, "I used a linear search" sounds like you did not know better. "I used a linear search, here is the note where I wrote down why, and here is the condition that would make me change it" is a completely different answer, and it is only available to you if you write it down now.

Checkpoint

$ make EXTRA_CFLAGS=-Werror test golden: 13 cases match all unit tests passed

Chapter six is types, where let x = 10; let y = true; print x + y; gets refused before it runs.

Golden tests pinned and the docs updated.

Statements and Variables

The difference between a thing and an event

A list that grows

Six new node kinds, and a field on every node

Three constructors

Free the new kinds

Give every node its position

A name as an expression

Parsing statements, and the assignment trick

The semicolon token, and where names live

Running a program instead of an expression

Repair the old tests, then add new ones

Pin the new errors, and write it down