Five new kinds of token: a name, three keywords, and the character =.
src/lexer.h · inside the typedef enum
TOK_RPAREN, TOK_EQUAL,The single character =. It goes here, with the other punctuation, because that is what it is.
TOK_IDENT,A name written in the source: x, age, total_count. It sits next to the keywords because a keyword is a name that the language reserved.
TOK_PRINT,print is a keyword rather than a function, which keeps the parser small: a statement becomes the word print followed by an expression. In print(x + y) the parentheses are then just ordinary grouping you already parse.
TOK_LET,let binds a name that cannot change afterwards.
TOK_VAR,var binds one that can. Immutable by default is a language design decision, made here, in this line.
TOK_EOF,Leave these two last. The rest of the compiler reads them as the pair meaning "not a real piece of syntax".
TOK_ERROR
An enum is just a list of names for numbers. TOK_RPAREN is
6, TOK_EQUAL is now 7, and so on. You will never care about the numbers.
What you care about is that the compiler can now tell you when you forget one.
Checkpoint
This is supposed to fail, with errors about
token_type_name not handling the new values. That is step 2's job, and
the compiler finding it for you is the entire point of step 2.
Somebody reads these strings when the parser says
expected ')', found IDENT. Write them like somebody is going to read them,
because somebody is.
src/lexer.c · inside token_type_name, before the TOK_ERROR case
case TOK_EQUAL:A case label. Execution jumps here when the switch value equals TOK_EQUAL.
return "EQUAL";Returning leaves the function immediately, which is why none of these cases needs a break.
case TOK_IDENT: return "IDENT"; case TOK_PRINT: return "PRINT"; case TOK_LET: return "LET"; case TOK_VAR: return "VAR";
That switch has no default: case and I left it out on purpose. A switch
over an enum with no default makes clang list every value you did not handle. Add a
default: and you lose that warning for the rest of the project's life,
quietly, without anyone noticing. Leave it out. Every time you add a token type from
here on, the compiler walks you to each place that has to deal with it.
Checkpoint
Clean build, every earlier test still passing. You added a whole token family and disturbed nothing.
Before the lexer can read a name it has to know which characters a name is made of.
Two small functions, shaped the same way as is_digit right above them.
src/lexer.c · directly below is_digit
static int is_ident_start(char c) {static makes this name private to lexer.c, so no other file can call it and no other file can collide with it. It returns int rather than bool to match is_digit, which set the pattern.
return (c >= 'a' && c <= 'z') ||Letters sit in order in the character set, so "is it a lowercase letter" really is two comparisons. && is and, || is or.
(c >= 'A' && c <= 'Z') || c == '_';Underscore counts as a letter. Nearly every language allows it, and _tmp is a convention people expect to be able to write.
} static int is_ident_part(char c) {The rule for every character after the first.
return is_ident_start(c) || is_digit(c);Calling the other two rather than repeating their comparisons. If you ever change what a name may start with, this follows automatically.
}
C already has isalpha in <ctype.h> and it looks like
it does exactly this. It does not. Its answer changes depending on the operating
system's locale setting, and under some locales it says yes to bytes above 127. That
would let stray UTF-8 into your names based on which machine happened to compile the
compiler. Your language should decide its own alphabet instead of borrowing the
operating system's.
Look at what you just wrote. The first character cannot be a digit, but every
character after it can. There is a real reason for that. If a name could start with a
digit then 2x would be a legal name, and the lexer would no longer be able
to tell what it is looking at from one character. That single character decision is
what keeps a lexer a simple loop. Pretty much every language draws the line in the same
spot.
Checkpoint
Plain make test this time, not the -Werror version. Nothing
calls these two yet, so clang warns that they are dead. The warning is correct and it
goes away in step 5. It is also why this checkpoint drops -Werror: under
that flag an unused function is a build failure.
This is the one with the real trap in it. Read the whole step before you type anything.
src/lexer.c · above lexer_next · needs #include <string.h> at the top of the file
static int word_is(const Lexer *lx, int start, const char *keyword) {Asks one question: is the text between start and where the lexer is now exactly this keyword? const Lexer *lx is a pointer the function promises not to write through.
int length = lx->pos - start;The word's length. lx->pos is where the scanner stopped and start is where the word began, so the difference is how many characters it ate.
if ((size_t)length != strlen(keyword)) {The whole trap lives on this line. Lengths first. strlen walks the keyword to its terminating zero byte and gives its length. The (size_t) cast is there because strlen returns an unsigned type, and comparing that to a signed int is a warning under -Wextra.
return 0;Different lengths, different words. Leave before touching a single byte.
} return memcmp(lx->src + start, keyword, (size_t)length) == 0;lx->src + start is the address of the word's first character. memcmp compares exactly length bytes and returns 0 when they are identical, so == 0 turns that into a yes.
}
You cannot use strcmp here and you cannot skip the length check, both
for the same reason. A token's text has no zero byte at the end of it.
A token is a pointer into the source plus a length. In the source
let x = 10, the text starting at l keeps right on going to
the end of the file. Nothing tells a string function where your word stops.
So strcmp would compare "let x = 10" against
"let" and read straight past your word into the rest of the file. And
memcmp without the length check compares three bytes of
letter against let, finds them identical, and says they
match. It never looks at byte four. Line 3 is what stops that.
Checkpoint
Still unused, still fine. Three dead functions waiting for step 5 to call them.
This is the middle of the chapter. Seven lines, and the order of the two halves is the whole idea.
src/lexer.c · inside lexer_next · after the is_digit block, before advance(lx);
if (is_ident_start(c)) {c is the character the scanner is sitting on and has not consumed yet. One test decides whether this whole block runs.
while (is_ident_part(peek(lx))) advance(lx);The munch. peek looks at the current character without moving; advance consumes it. This eats the first character too, since anything passing is_ident_start also passes is_ident_part. Nothing is accumulated, unlike the number branch above, because the text already sitting in the source is the answer.
if (word_is(lx, start, "let")) return make_token(lx, TOK_LET, start);Only now, with the word finished, do you ask what it was. make_token stamps in the type and works out the pointer and length from start.
if (word_is(lx, start, "var")) return make_token(lx, TOK_VAR, start); if (word_is(lx, start, "print")) return make_token(lx, TOK_PRINT, start);Three keywords, three ifs. That is enough. Real compilers switch to a hash table somewhere past a dozen of them.
return make_token(lx, TOK_IDENT, start);Matched nothing, so it is an ordinary name. This line is what makes every keyword a name first.
}
The obvious way to do this is to check while you scan. See l, peek for
e, peek for t, hand back the keyword. Then you hit the word
letter and it falls apart. Checking as you go, you hand back
LET and then a separate name ter. One variable quietly
became two tokens, and you find out about it somewhere else entirely.
So eat the longest run of name characters you can, and only then work out what it
was. letter eats all six characters, fails word_is on
length, and stays a name. The rule has a name, maximal munch, and every real lexer
follows it.
You never store the name anywhere. A Token already carries a pointer into
the source and a length, and those two fields together are the name. Nothing
to allocate, nothing to free. t.value stays zero, since it only means
anything for a number.
Checkpoint
The unused-function warnings are gone, because something finally calls all three. You now have a lexer that reads names.
The easiest line in the chapter.
src/lexer.c · in the switch (c) at the bottom of lexer_next
case ')': return make_token(lx, TOK_RPAREN, start); case '=': return make_token(lx, TOK_EQUAL, start);Same shape as the six above it. This switch runs after advance(lx), so the character is already consumed and make_token sees a length of one.
Something for later that you should not build today. When comparisons show up
you will need == too, and this case will have to peek at the next character
to tell assignment apart from equality. It is a two line change to this one case when
you get there. Writing it now gives you code with no test and nothing calling it.
Checkpoint
This file is yours to write. The Makefile finds tests by filename, so there is nothing to wire up.
tests/test_lexer_5.c · new file · copy the shape of tests/test_lexer_4.c
Write one helper first, since you will want it in every single assertion. Given a token
and a string, is the token's text exactly that string? Same trap as step 4. Compare
t.length against strlen first, then memcmp. A
helper that skips the length check happily reports that letter matches
let. A test that lies to you is worse than having no test.
Then these, in this order. Each catches more than the last.
"x"One IDENT token, length 1, text x, then EOF. Assert the text, not just the type. Checking only the type passes even when your loop consumed the wrong number of characters.
"total_count2"One token, length 12. Proves digits and underscores are accepted after the first character.
"let var print"Three keyword types in order, then EOF. The happy path.
"letter"The one that matters. One IDENT with text letter, and the next token is EOF. With eager classification, the EOF assertion is what fails. Do the same for variable and printer, since each attacks a different keyword.
"lets" "_let" "Let"All three are names, not keywords. Writing that third one is you deciding the language is case sensitive. Write that down somewhere.
"=" then "=="One EQUAL, then two EQUALs. Two is correct today. Pinning it now means the test fails loudly when you add real == support, instead of the change slipping past unnoticed.
"let x = 10\nlet y = 20\nprint(x + y)"The whole milestone in one pass. Build an array of expected types and loop over it. Keep the newlines in: this also proves they are still skipped as plain whitespace.
Checkpoint
No code in this one. This is a decision the next chapter cannot start without, and it is expensive to change your mind about later.
That last assertion just locked in that the language treats a newline as ordinary whitespace. So what tells the parser one statement has finished and the next one has started? There are three answers and real languages use all three.
let x = 10;A required semicolon. One token of work in the parser. Costs a character of typing per line. C, Java and Rust do this.
let x = 10↵The newline ends it. Your lexer stops treating newlines as whitespace and starts carrying line structure, and every later phase inherits that. Python and Go do this, and Go needed a rule that inserts semicolons for you to make it work.
let x = 10 let y = 20Nothing at all. Works today, because every statement starts with a keyword. Stops working as soon as a bare expression becomes a legal statement, and the parser can no longer tell where one ends.
I went with A. It costs you one character per line. B costs you extra complexity in every phase after this one. C costs you a breaking change to a language that already has programs written in it.
Checkpoint